Papers with multi-task learning
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (D19-58)
Copied to clipboard
| Challenge: | MRQA datasets have been used to benchmark progress in general-purpose language understanding. |
| Approach: | They propose to combine 18 question answering datasets into one shared task to evaluate their generalization capabilities. |
| Outcome: | The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than baseline based on BERT. |
Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priors (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to facilitate distantly supervised relation extraction are noisy instances, long-tail relations and unbalanced bag sizes. |
| Approach: | They propose a multi-task approach to facilitate distantly supervised relation extraction by bringing closer the representations of sentences that contain the same Knowledge Base pairs. |
| Outcome: | The proposed approach improves performance on two datasets created via distant supervision. |
Practical Semantic Parsing for Spoken Language Understanding (N19-2)
Copied to clipboard
| Challenge: | Existing systems that can handle a user's utterance are unable to handle Q&A or SLU. |
| Approach: | They build a transfer learning framework for executable semantic parsing . they show it is effective for Q&A and for spoken language understanding . |
| Outcome: | The proposed framework is effective for Q&A and Spoken Language Understanding . it can be learned by exploiting data on other domains, the authors show . |
They Exist! Introducing Plural Mentions to Coreference Resolution and Entity Linking (C18-1)
Copied to clipboard
| Challenge: | Unlike singular mentions each of which represents one entity, plural mentions stand for multiple entities. |
| Approach: | They propose a novel coreference resolution algorithm that selectively creates clusters to handle both singular and plural mentions and a deep learning-based entity linking model that jointly handles both types of mentions through multi-task learning. |
| Outcome: | The proposed model outperforms existing models designed for singular mentions and plural mentions. |
Towards Unifying the Label Space for Aspect- and Sentence-based Sentiment Analysis (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train ABSA model are limited by lack of annotated data . a dual-granularity pseudo labeling approach is proposed to solve this problem . |
| Approach: | They propose a framework for aspect-based sentiment analysis that uses annotated data to train ABSA models. |
| Outcome: | The proposed framework surpasses previous methods on benchmarks. |
Toward Building a Language Model for Understanding Temporal Commonsense (2022.aacl-srw)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT are still poor in temporal reasoning . commonsense reasoning is crucial for natural language processing (NLP) |
| Approach: | They propose to use multi-step fine-tuning and masked language modeling to predict mangled temporal indicators that are crucial for commonsense reasoning. |
| Outcome: | The proposed model improves performance on multiple time-related tasks. |
Building a Dialogue Corpus Annotated with Expressed and Experienced Emotions (2022.acl-srw)
Copied to clipboard
| Challenge: | a human would recognize the emotion of an interlocutor and respond with an appropriate emotion, such as empathy and comfort. |
| Approach: | They propose to build a dialogue corpus annotated with two kinds of emotions . they collect tweets and annotate them with the emotion they put into the utterance . |
| Outcome: | The proposed method shows that it is difficult to recognize experienced emotions and multitask learning is effective. |
Multi-Domain Multilingual Question Answering (2021.emnlp-tutorials)
Copied to clipboard
| Challenge: | Question answering (QA) is one of the most challenging tasks in natural language processing. |
| Approach: | a tutorial examines the state-of-the-art approaches to multi-domain and multilingual QA . they introduce standard benchmarks and discuss out-of the-box training with open-domain QA systems . |
| Outcome: | This tutorial aims to bridge the gap between open-domain and multilingual QA. |
Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework . |
| Approach: | They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations. |
| Outcome: | The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size. |
A Frustratingly Easy Approach for Entity and Relation Extraction (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing work on end-to-end relation extraction models combine two tasks: named entity recognition and relation extraction. |
| Approach: | They propose a pipelined approach for entity and relation extraction that uses two independent encoders to construct the relation model. |
| Outcome: | The proposed approach achieves an 8.16 speedup with a slight reduction in accuracy on standard benchmarks. |
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020.aacl-demo)
Copied to clipboard
| Challenge: | End-to-end sequence-to sequence (S2S) modeling has witnessed rapid growth in speech-totext (ST) tasks. |
| Approach: | They introduce fairseq S2T, a fairsq extension for speech-to-text modeling tasks such as end-to end speech recognition and speech-text translation. |
| Outcome: | The proposed extension provides end-to-end workflows from data pre-processing, model training to offline (online) inference. |
Multi-Task Learning for Argumentation Mining in Low-Resource Settings (N18-2)
Copied to clipboard
| Challenge: | Argument component identification is difficult for trained annotators to perform in a new domain or to develop new AM tasks. |
| Approach: | They investigate whether multi-task learning can improve performance on AM problems . they found that MTL performs particularly well when little training data is available for the main task . |
| Outcome: | The proposed approach performs better when little training data is available for the main task, a common scenario in AM. |
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models for story generation suffer from repetition, logic conflicts, and lack of long-range coherence . |
| Approach: | They propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories by multi-task learning. |
| Outcome: | The proposed model can generate more reasonable stories than state-of-the-art models, compared with existing models, showing that it can capture useful semantic and syntactic features. |
SEPSIS: I Can Catch Your Lies – A New Paradigm for Deception Detection (2025.acl-srw)
Copied to clipboard
Anku Rani, Dwip Dalal, Shreya Gautam, Pankaj Gupta, Vinija Jain, Aman Chadha, Amit Sheth, Amitava Das
| Challenge: | a new framework categorizes deception into three forms: lies of omission, lies of commission, and lies of influence . a novel framework for deception detection leveraging NLP techniques is proposed . |
| Approach: | They propose a framework that categorizes deception into three forms: lies of omission, lies of commission, and lies of influence. |
| Outcome: | The proposed framework achieves an impressive F1 score of 0.87 across all layers . it can be used to investigate lies of omission, lies of commission and lies of influence . |
Multi-Task, Multi-Channel, Multi-Input Learning for Mental Illness Detection using Social Media Text (D19-62)
Copied to clipboard
| Challenge: | Existing methods for mental illness detection have limited data available for training . lack of sufficient annotated data and inability to extract explanations on the derived outcome have restricted researchers to use traditional methods. |
| Approach: | They propose to use emotional patterns identified by clinical practitioners to enhance the prediction capabilities of a mental illness detection model built using a deep neural network architecture. |
| Outcome: | The proposed method achieves a task-specific AUC higher than 0.90 . it compares multi-task learning with multi-channel convolutional neural network and multiple inputs to methods such as multi-class classification . |
Multi-Surrogate-Objective Optimization for Neural Topic Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Neural topic modeling incorporates multiple loss functions but can be difficult to optimize for disparate magnitudes of these losses. |
| Approach: | They propose a gradient-based multi-objective optimization approach that integrates MOO algorithms into the model without the need for hard-parameter sharing. |
| Outcome: | The proposed approach outperforms direct MOO applications on NTMs. |
Trigger Word Detection and Thematic Role Identification via BERT and Multitask Learning (D19-57)
Copied to clipboard
| Challenge: | Using natural language processing to discover and mine drug-related knowledge from text has been a hot topic in recent years. |
| Approach: | They propose to use a pre-trained biomedical language representation model to extract mutation-disease knowledge from PubMed. |
| Outcome: | The proposed approaches achieve 0.60 (ranks 1) and 0.25 (rank 2) on task 1 and task 2 respectively in terms of F1 metric. |
Multi-task Learning of Spoken Language Understanding by Integrating N-Best Hypotheses with Hierarchical Attention (2020.coling-industry)
Copied to clipboard
| Challenge: | Existing methods to integrate hypotheses into speech recognition systems are noisy and can cause information loss. |
| Approach: | They propose to integrate hypotheses into multi-task learning and transfer learning to improve performance. |
| Outcome: | The proposed model improves domain and intent classification by 19% and 37% compared to current methods . the proposed model could recover transcription and rewrite the query for a better understanding . |
Towards Unification of Discourse Annotation Frameworks (2022.acl-srw)
Copied to clipboard
| Challenge: | Discourse information is difficult to represent and annotate, and corpora annotated under different frameworks vary considerably. |
| Approach: | They propose to use automatic means to unify discourse structures and relations . they will also explore the application of the unified framework in multi-task learning and graphical models . |
| Outcome: | The proposed method can be used in multi-task learning and graphical models. |
Few-Shot and Zero-Shot Learning for Historical Text Normalization (D19-61)
Copied to clipboard
| Challenge: | Historical text normalization often relies on small training datasets. |
| Approach: | They evaluate 63 multi-task learning configurations for sequence-to-sequence-based historical text normalization across ten datasets from eight languages. |
| Outcome: | The proposed learning architecture outperforms the simple, but strong identity baseline. |
An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models for named entity recognition only consider the potential transferability between two identical tasks across both domains. |
| Approach: | They propose to use a similarity metric model to improve cross-lingual named entity recognition task on target domain. |
| Outcome: | Empirical studies on 7 different languages confirm the effectiveness of the proposed model. |
Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports (2022.naacl-industry)
Copied to clipboard
Arijit Sehanobish, McCullen Sandora, Nabila Abraham, Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker, Richard Herzog, Benjamin Odry, Ron Vianu
| Challenge: | Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP but training or fine-tuning these models for individual tasks can be time consuming and resource intensive. |
| Approach: | They propose to use pretrained Transformer based models finetuned on domain specific corpora to train models for individual tasks. |
| Outcome: | The proposed model can match the performance of a task specific model when the task specific models show similar representations across all of their hidden layers and their gradients are aligned, i.e. their gradient follows the same direction. |
Phylogenic Multi-Lingual Dependency Parsing (N19-1)
Copied to clipboard
| Challenge: | phylogenetic learning is beneficial for low resourced languages and well furnished languages families. |
| Approach: | They propose to use the phylogenetic tree to guide the learning of multi-lingual dependency parsers . they use a phylogy tree to train models that leverage languages structural similarities . |
| Outcome: | The proposed model outperforms independently learned models on zero-shot parsing of unseen languages. |
Multi-task Learning for Automated Essay Scoring with Sentiment Analysis (2020.aacl-srw)
Copied to clipboard
| Challenge: | Automated Essay Scoring (AES) is a process that aims to alleviate the workload of graders and improve the feedback cycle in educational systems. |
| Approach: | They propose to combine two tasks, sentiment analysis and AES by utilizing multi-task learning to combine sentiment features extracted from opinion expressions. |
| Outcome: | The proposed model produces a QWK of 0.763 on the Automated StudentAssessment Prize (ASAP) benchmark. |
Towards Reliable and Practical Phishing Detection (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing datasets lack size and diversity, with only 609 voice phishing samples available in Korean and 638 smishing instances available in English. |
| Approach: | They propose to use a Korean dataset to construct a reliable phishing detection system using language models to evaluate the model's in-domain and unseen attack detection performance. |
| Outcome: | The proposed system performs reasonably well in voice and unseen attacks while smishing detection remains challenging. |
Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP (2021.eacl-demos)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) has become a standard repertoire in natural language processing (NLP) it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters. |
| Approach: | They propose a toolkit for fine-tuning contextualized embeddings in multi-task settings. |
| Outcome: | The proposed toolkit supports a variety of natural language processing tasks . it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters. |
Named Entity Recognition in Multi-level Contexts (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition are unsatisfactory for recognizing entities in limited or ambiguous sentence-level contexts. |
| Approach: | They propose a framework to incorporate multi-level contexts for named entity recognition using TagLM as a baseline model and an auxiliary task to mine word-level contextual information. |
| Outcome: | The proposed framework is based on a set of sentence-level contexts and a document-level task to mine word-level contextual information. |
Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining (2022.naacl-industry)
Copied to clipboard
| Challenge: | Existing approaches to clustering unlabeled utterances are based on transformerbased sentence embedding methods. |
| Approach: | They propose a semantic embedding approach that can be leveraged to identify clusters of utterances that correspond to unhandled intents. |
| Outcome: | The proposed approach can identify clusters of utterances that correspond to unhandled intents from a large collection of enterprise virtual assistant data using a multi-task softmax loss. |
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding (D19-58)
Copied to clipboard
| Challenge: | In-domain datasets are used to train and validate our model, and other out-of-domain data are used for validation. |
| Approach: | They propose a model which uses cross-task learning with expert representation for the generalization of reading and understanding. |
| Outcome: | The proposed model achieved an average F1 score of 66.1 % in the out-of-domain setting, which is a 4.3 percentage point improvement over the official BERT baseline model. |
T2NER: Transformers based Transfer Learning Framework for Named Entity Recognition (2021.eacl-demos)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is an important task in information extraction due to large variations in entity names and flexibility in how entities are mentioned. |
| Approach: | They propose a Transformers based Transfer Learning framework for Named Entity Recognition (T2NER) that integrates transformer models with the state-of-the-art in NLP and provides a unified platform for transfer learning. |
| Outcome: | The proposed framework bridges the gap between the state-of-the-art in transformer models and the state of the art in NER with deep transformer models. |
Retrieval, Re-ranking and Multi-task Learning for Knowledge-Base Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing work on question answering over knowledge bases limited the search space to a subset of KBs . a retrieval-and-rerank framework is used to access KB and rerank retrieved candidates with more powerful neural networks. |
| Approach: | They propose to share a BERT encoder across all three sub-tasks and define task-specific layers on top of the shared layer. |
| Outcome: | The proposed method improves accuracy and accuracy on the SimpleQuestions dataset and the FreebaseQA dataset. |
A Match Made in Heaven: A Multi-task Framework for Hyperbole and Metaphor Detection (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to detect metaphor and hyperbole independently have not explored their relationship computationally. |
| Approach: | They propose a multi-task deep learning framework to detect hyperbole and metaphor simultaneously by annotating two hyperbolic datasets with metaphor labels. |
| Outcome: | The proposed framework improves state-of-the-art hyperbole detection by 12% over existing methods. |
Event Causality Identification via Generation of Important Context Words (2022.starsem-1)
Copied to clipboard
| Challenge: | Prior work focused on identifying causal relation between two event mentions . current models do not output important contexts for causal prediction of two mentions. |
| Approach: | They propose to use dependency path generation as a complementary task for ECI. |
| Outcome: | The proposed model can generate both causal relation and dependency path words from input sentences. |
D-NET: A Pre-Training and Fine-Tuning Framework for Improving the Generalization of Machine Reading Comprehension (D19-58)
Copied to clipboard
Hongyu Li, Xiyuan Zhang, Yibing Liu, Yiming Zhang, Quan Wang, Xiangyang Zhou, Jing Liu, Hua Wu, Haifeng Wang
| Challenge: | MRC requires machines to understand text and answer questions about the text. |
| Approach: | They propose a simple system Baidu submitted for MRQA 2019 Shared Task that focused on generalization of machine reading comprehension (MRC) models. |
| Outcome: | The proposed system is ranked at top 1 of all participants in terms of averaged F1 score. |
When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Transfer learning (TL) in natural language processing has seen a surge of interest in recent years . pre-trained models have shown impressive ability to transfer to novel tasks . |
| Approach: | They compare two different methods of transfer learning in natural language processing to find out which is better. |
| Outcome: | The proposed methods perform better when the target task has fewer instances than the supporting task and vice versa. |
Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Existing tools for fine-grained human evaluation lack adaptability to different domains or languages, or modify annotation settings according to user needs. |
| Approach: | They propose a unified platform for fine-grained evaluation that is customizable and deployable with a single YAML configuration file. |
| Outcome: | The proposed frameworks are based on a single YAML configuration file and can be easily extended to different domains or languages. |
Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on syntactic injection in Variational AutoEncoders (VAEs) are limited to LSTM-based VAEs. |
| Approach: | They propose to use latent space separation techniques to inject syntactic information into Variational AutoEncoders (VAEs) using graph-based models. |
| Outcome: | The proposed end-to-end VAE architecture can improve the organisation of the latent space, alleviating the information loss occurring in standard VAE setups, and resulting in enhanced performances on language modelling and downstream generation tasks. |
Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multi-task auxiliary learning uses a set of relevant auxiliary tasks to improve performance of a primary task. |
| Approach: | They propose a time-efficient sampling method to select the most beneficial sub-datasets from the auxiliary tasks to achieve efficient multi-task auxiliary learning. |
| Outcome: | The proposed method significantly outperforms random sampling and ST-DNN on three benchmark datasets. |
Paper Bullets: Modeling Propaganda with the Help of Metaphor (2023.findings-eacl)
Copied to clipboard
| Challenge: | We hypothesize that it can be beneficial to model propaganda and metaphor together . we identify propaganda using loaded language and name-calling . |
| Approach: | They hypothesize that it can be beneficial to model propaganda and metaphor together . they use two datasets to identify propaganda techniques in news articles and memes . |
| Outcome: | The proposed model improves performance for the two most common propaganda techniques, especially loaded language and name-calling. |
R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-task Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) improves performance in multi-task learning by diversifying the head matrices through Multi-Head Dropout and Multi-head Random Initialization. |
| Approach: | They propose a low-rank adaptive approach to fine-tune large language models by approximating weight updates through low-ranked matrices. |
| Outcome: | The proposed approach improves performance in multi-task learning while reducing memory usage and training time. |
RACQC: Advanced Retrieval-Augmented Generation for Chinese Query Correction (2025.findings-emnlp)
Copied to clipboard
Jinbo Su, Lingzhe Gao, Wei Li, Shihao Liu, Haojie Lei, Xinyi Wang, Yuanzhao Guo, Ke Wang, Daiting Shi, Dawei Yin
| Challenge: | Large language models (LLMs) exhibit remarkable capabilities across many tasks, but face critical challenges in the CSC scenario: (1) poor generalization to rare entities in open-domain searches; and (2) failure to adapt to temporal entity variations due to static parameters, resulting in serious over-correction issues. |
| Approach: | They propose a Chinese Spelling Check system with RAG and multi-task learning that integrates dynamic knowledge retrieval and entity-centric RAG to address rare entities. |
| Outcome: | The proposed system outperforms existing baselines in the CSC task and achieves a maximum improvement of +9.92% on the search scenario benchmark and +3.2% on the general-domain dataset. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
Multi-doc Hybrid Summarization via Salient Representation Learning (2023.acl-industry)
Copied to clipboard
| Challenge: | Multi-document summarization is gaining more and more attention . extractive multi-doc approaches intend to directly extract key facts from multiple sources . |
| Approach: | They propose a multi-document hybrid summarization approach that generates a human-readable summary and extracts corresponding key evidences based on multi-doc inputs. |
| Outcome: | The proposed method generates a human-readable summary and extracts key evidences based on multi-doc inputs. |
PARSQL: Enhancing Text-to-SQL through SQL Parsing and Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models have made significant strides in text-to-SQL tasks, but small language models struggle to accurately interpret natural language questions due to resource limitations. |
| Approach: | They propose a SQL parser that extracts constraints from SQL to generate sub-SQLs . they use a rule-based and LLM-based method to generate step-by-step SQL explanations based on the results . |
| Outcome: | The proposed framework outperforms models with the same model size on BIRD and Spider benchmarks. |
Sluice Resolution without Hand-Crafted Features over Brittle Syntax Trees (N18-2)
Copied to clipboard
| Challenge: | sluice resolution in english is the problem of finding antecedents of wh-fronted ellipses . previous work relied on hand-crafted features over syntax trees that scale poorly to other languages and domains . |
| Approach: | They propose a model that uses partial parsing to find antecedents of wh-fronted ellipses in english . their model significantly outperforms previous work on available newswires . |
| Outcome: | The proposed model outperforms the only previous work on available newswires. |
Efficient Meta Lifelong-Learning with Limited Memory (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing natural language learning models fail to continuously learn new tasks as they are re-trained throughout their lifetime. |
| Approach: | They propose a meta-lifelong framework that combines three common lifelong learning principles . they propose to store past examples in episodic memory and replay them at training and inference time . |
| Outcome: | The proposed framework achieves state-of-the-art performance using 1% memory size and narrows the gap with multi-task learning. |
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating knowledge from multiple tasks suffer from catastrophic forgetting and difficulties in dataset balancing. |
| Approach: | They propose an algorithm that extracts and combine adapters in a knowledge composition step. |
| Outcome: | The proposed class outperforms traditional methods such as full fine-tuning and multi-task learning on 16 diverse NLU tasks. |
Dynamic Multi-Level Multi-Task Learning for Sentence Simplification (C18-1)
Copied to clipboard
| Challenge: | Sentence simplification is the task of improving readability and understandability of an input text. |
| Approach: | They propose a strong pointer-copy mechanism based sequence-to-sequence sentence simplification model and a novel ‘multi-level’ soft sharing approach where each auxiliary task shares different (higher versus lower) level layers of the model. |
| Outcome: | The proposed model outperforms competing simplification systems in SARI and FKGL automatic metrics, and human evaluation. |
English-to-Chinese Transliteration with Phonetic Auxiliary Task (2020.aacl-main)
Copied to clipboard
| Challenge: | Several NMT techniques have been used to enhance machine transliteration models, but few focus on the linguistic features specific to the relevant languages. |
| Approach: | They propose a phonetic auxiliary task that integrates phonetic features into a model to improve generalization performance of the main transliteration task. |
| Outcome: | The proposed model achieves similar performance to the current state of the art with a much smaller size. |
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that pre-trained language models perform poorly on challenging datasets where spurious correlations do not hold. |
| Approach: | They propose to use multi-task learning to improve generalization from minority examples . they propose to combine MTL with auxiliary tasks to improve performance . |
| Outcome: | The proposed model generalizes from minority examples without hurting in-distribution performance. |
Towards Generative Event Factuality Prediction (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for event factuality prediction focus on author's presentation of factuity . a novel end-to-end generative task is proposed to predict event factuality holders, targets, and their associated factual values. |
| Approach: | They propose a generative task and system for predicting event factuality holders, targets, and their associated factual values. |
| Outcome: | The proposed system improves on the FactBank corpus and other corpora . it can predict presentation of factuality of nested sources alongside their target events . |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
Modeling Local Contexts for Joint Dialogue Act Recognition and Sentiment Classification with Bi-channel Dynamic Convolutions (2020.coling-main)
Copied to clipboard
| Challenge: | a novel context-aware dynamic convolution network is proposed to better leverage the local contexts when dynamically generating convolution kernels. |
| Approach: | They propose a dynamic convolution network to leverage local contexts when generating convolution kernels. |
| Outcome: | The proposed frameworks achieve state-of-the-art on two benchmark datasets. |
SRL4ORL: Improving Opinion Role Labeling Using Multi-Task Learning with Semantic Role Labeling (N18-1)
Copied to clipboard
| Challenge: | Recent neural approaches do not outperform the state-of-the-art feature-based models for Opinion Role Labeling (ORL). |
| Approach: | They propose to use multi-task learning to improve Opinion Role Labeling by using a related task which has substantially more data. |
| Outcome: | The proposed model outperforms the state-of-the-art model for Opinion Role Labeling (ORL) with more data. |
Compositional Zero-Shot Domain Transfer with Text-to-Text Models (2023.tacl-1)
Copied to clipboard
Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-García, Naoto Usuyama, Sheng Zhang, Tristan Naumann, Aditya Nori, Hoifung Poon, Javier Alvarez-Valle, Ozan Oktay, Stephanie L. Hyland
| Challenge: | Existing approaches to zero-shot domain transfer are limited by domain gap and lack of in-domain labels. |
| Approach: | They propose a compositional transfer learning framework (DoT51) that learns domain knowledge and task knowledge in a multi-task manner without access to in-domain labels. |
| Outcome: | The proposed framework outperforms the current state-of-the-art in zero-shot domain transfer by over 7 absolute points in accuracy on RadNLI. |
Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation (P18-1)
Copied to clipboard
| Challenge: | Recent advances on abstractive summarization have allowed substantial improvements in the quality of the model, but there is still scope for improvement. |
| Approach: | They propose novel multi-task architectures with high-level layer-specific sharing across multiple encoder and decoder layers of the three tasks and soft-sharing mechanisms. |
| Outcome: | The proposed model improves on the CNN/DailyMail and Gigaword datasets and on the DUC-2002 transfer setup. |
HyperIDP: Customizing Temporal Hypergraph Neural Networks for Multi-Scale Information Diffusion Prediction (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on information diffusion prediction have focused on both macroscopic and microscopic scales. |
| Approach: | They propose a hypergraph-based model that manages both macroscopic and microscopic diffusion predictions. |
| Outcome: | The proposed model outperforms baseline models on both macroscopic and microscopic tasks. |
Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning (2022.acl-short)
Copied to clipboard
| Challenge: | Large multilingual pretrained language models such as mBERT and XLM-RoBERTa have been found to be effective for cross-lingual transfer of syntactic parsing models but only between related languages. |
| Approach: | They propose to use multi-task learning to dynamically optimize for parsing performance on outlier languages by using a multi-level learning approach. |
| Outcome: | The proposed method significantly outperforms uniform and size-proportional sampling in the zero-shot setting. |
A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods (2023.eacl-main)
Copied to clipboard
| Challenge: | Multi-task learning is a popular approach in natural language processing because of its commonalities and differences. |
| Approach: | They propose to summarize recent advances in multi-task learning methods based on their task relatedness into two general multi-step training methods. |
| Outcome: | The proposed methods summarize the tasks and discuss future directions. |
T 2 -NER: A Two-Stage Span-Based Framework for Unified Named Entity Recognition with Templates (2023.tacl-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) has evolved from flat to overlapped and discontinuous . NER is a text recognition task that recognizes mentions that represent entities in text . |
| Approach: | They propose a two-stage span-based framework to solve a unified NER task using two stages . they extract entity spans, classify over all entity span pairs and combine them to train two stages. |
| Outcome: | The proposed framework beats all the current competitive baselines on eight benchmark datasets, obtaining the best performance of unified NER. |
XLPT-AMR: Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) research is limited and challenging for languages other than English. |
| Approach: | They propose a cross-lingual pre-training approach for AMR parsing and text generation . they use an English-to-English parallel dataset and a multi-task learning approach . |
| Outcome: | The proposed approach outperforms baseline pre-training methods on English parsing and text generation tasks. |
Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation (2025.findings-acl)
Copied to clipboard
Yuhao Zhang, Xiangnan Ma, Kaiqi Kou, Peizhuo Liu, Weiqiao Shan, Benyou Wang, Tong Xiao, Yuxin Huang, Zhengtao Yu, JingBo Zhu
| Challenge: | Existing textless speech-to-speech translation models have two main challenges: 1) learning cross-modal features and 2) learning alignment of difference languages in long sequences. |
| Approach: | They propose a unit language to overcome two main modeling challenges . they propose task prompt modeling to utilize the unit language in guiding the modeling process. |
| Outcome: | The proposed language improves over a strong baseline and achieves comparable performance to models trained with text. |
Multi-task Learning for Multilingual Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual neural machine translation systems rely on bitext training data, which is limited and costly to collect. |
| Approach: | They propose a multi-task learning framework that trains the model with the translation task on bitext data and two denoising tasks on monolingual data. |
| Outcome: | The proposed framework outperforms pre-training models for both NMT and cross-lingual transfer learning NLU tasks. |
A Distance-Aware Multi-Task Framework for Conversational Discourse Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have focused on graph-based and transition-based discourse parsing, but no study has investigated the advantages of both paradigms for conversational discourse paring. |
| Approach: | They propose a distance-aware multi-task framework that incorporates the strengths of transition-based paradigms to facilitate conversational discourse parsing. |
| Outcome: | The proposed framework improves the graph-based paradigm on long-distance dependency links. |
I am PsyAM: Modeling Happiness with Cognitive Appraisal Dimensions (2023.findings-acl)
Copied to clipboard
| Challenge: | Emotions are an indicator of psychological states, such as happiness, which can be modelled through cognitive appraisal theory (CAT) |
| Approach: | They propose to use adaptor modules in a sequential multi-task learning setup to generate high-dimensional feature representations of hedonic well-being (momentary happiness) they propose to apply feature fusion methods to model emotion in text . |
| Outcome: | The proposed framework has cross-task validity and generalizability and is robust against traditional methods and BERT baselines. |
Multi-Task Networks with Universe, Group, and Task Feature Learning (P19-1)
Copied to clipboard
| Challenge: | In multi-task learning, multiple related tasks are learned together. |
| Approach: | They propose methods that take advantage of natural groupings of related tasks . they propose parallel and serial architectures that can learn different feature spaces . |
| Outcome: | The proposed methods improve performance on natural language understanding (NLU) tasks. |
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have limited generalization capabilities and performance challenges. |
| Approach: | They evaluate 15 different backbone LLMs and non-LLMs to evaluate their performance . larger models and extensive pre-training consistently enhance in-domain accuracy and data efficiency . |
| Outcome: | The results show that larger models and extensive pre-training enhance in-domain accuracy and data efficiency. |
ReasVQA: Advancing VideoQA with Imperfect Reasoning Process (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to VideoQA often fail when complex reasoning or temporal relationships are involved. |
| Approach: | They propose a method that leverages reasoning processes generated by Multimodal Large Language Models to improve VideoQA models. |
| Outcome: | The proposed method improves VideoQA models on three benchmarks. |
Database Search Results Disambiguation for Task-Oriented Dialog Systems (2022.naacl-main)
Copied to clipboard
Kun Qian, Satwik Kottur, Ahmad Beirami, Shahin Shayandeh, Paul Crook, Alborz Geramifard, Zhou Yu, Chinnadhurai Sankar
| Challenge: | Task-oriented dialog systems can't handle multiplesearch results when querying a database due to the lack of such scenarios in existing datasets. |
| Approach: | They propose a task that focuses on disambiguating database search results by synthetically generating turns through a pre-defined grammar and collecting human paraphrases for a subset. |
| Outcome: | The proposed task improves performance on DSR-disambiguation even in the absence of in-domain data, suggesting it can be learned as a universal dialog skill. |
Multi-Task Neural Models for Translating Between Styles Within and Across Languages (C18-1)
Copied to clipboard
| Challenge: | Generating natural language requires conveying content in an appropriate style. |
| Approach: | They propose to solve two related tasks on generating text of varying formality jointly using multi-task learning. |
| Outcome: | The proposed models achieve state-of-the-art performance for formality transfer and formality-sensitive machine translation without training on style-annotated translation examples. |
Dynamic Sampling Strategies for Multi-Task Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | Prior work focused on model architecture or generalization to held out datasets and largely passed over the particulars of the multi-task learning set up. |
| Approach: | They propose a dynamic sampling strategy that selects instances proportional to the model's current performance on a dataset relative to its single task performance. |
| Outcome: | The proposed model outperforms the best model on ORB, a recent multitask reading comprehension benchmark. |
Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding (2023.eacl-main)
Copied to clipboard
| Challenge: | State-of-the-art entity tracking approaches either design complicated model architectures or rely on task-specific pre-training to achieve good results. |
| Approach: | They propose a multi-task learning-enabled entity tracking approach that utilizes knowledge gained from general domain tasks to improve entity tracking. |
| Outcome: | The proposed approach achieves state-of-the-art on two popular entity tracking datasets, even though it does not require any task-specific architecture design or pre-training. |
Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs? (2026.eacl-long)
Copied to clipboard
Kai Sun, Yin Huang, Srishti Mehra, Mohammad Kachuee, Xilun Chen, Renjie Tao, Zhaojiang Lin, Andrea Jessee, Nirav Shah, Alex L Betty, Yue Liu, Anuj Kumar, Wen-tau Yih, Xin Luna Dong
| Challenge: | Existing literature on knowledge extraction for question answering questions whether it is still relevant for question answerrs. |
| Approach: | They extend an existing benchmark with knowledge extraction annotations and evaluate commercial and open-source LLMs of varying sizes. |
| Outcome: | The proposed model can achieve high QA accuracy, but can still benefit from knowledge extraction through augmentation with extracted triples and multi-task learning. |
Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts (P19-1)
Copied to clipboard
| Challenge: | Emotion cause extraction (ECE) aims at extracting potential causes behind certain emotions in text. |
| Approach: | They propose a 2-step task to extract potential pairs of emotions and corresponding causes in a document. |
| Outcome: | The proposed task is based on a benchmark emotion cause corpus. |
Detection-Correction Structure via General Language Model for Grammatical Error Correction (2024.acl-long)
Copied to clipboard
| Challenge: | Grammatical error correction (GEC) is a task dedicated to rectifying texts with minimal edits. |
| Approach: | They propose a detection-correction structure based on the general language model which integrates detection and correction into a single model. |
| Outcome: | The proposed model outperforms the state-of-the-art models on English and Chinese datasets. |
Multi-task Learning for Paraphrase Generation With Keyword and Part-of-Speech Reconstruction (2022.findings-acl)
Copied to clipboard
| Challenge: | PGKPR is a deep learning approach to generate paraphrases using key semantics of the source sentence. |
| Approach: | They propose a model with keyword and part-of-speech reconstruction for paraphrase generation using deep learning. |
| Outcome: | The proposed model outperforms comparative models on two commonly-used datasets. |
Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning fail to capture label correlations . Existing methods suffer from label order dependency, label combination over-fitting and error propagation problems. |
| Approach: | They propose a novel approach with multi-task learning to enhance label correlation feedback. |
| Outcome: | The proposed method outperforms baselines on AAPD and RCV1-V2 datasets. |
Strong Baselines for Complex Word Identification across Multiple Languages (N19-1)
Copied to clipboard
Pierre Finnimore, Elisabeth Fritzsch, Daniel King, Alison Sneyd, Aneeq Ur Rehman, Fernando Alva-Manchego, Andreas Vlachos
| Challenge: | Complex Word Identification (CWI) is the task of identifying which words or phrases in a sentence are difficult to understand by a specific type of reader. |
| Approach: | They propose to use monolingual and cross-lingual CWI models to make predictions for languages not seen during training. |
| Outcome: | The proposed models perform as well as (or better than) most models submitted to the latest CWI Shared Task. |
MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods face the trilemma of performance, data privacy, and computational costs, which hinders their application to LLMs. |
| Approach: | They propose a model-exclusive task arithmetic method for merging GPT-scale models which is data-agnostic and bypasses the heavy search process. |
| Outcome: | The proposed method achieves state-of-the-art performance on multiple tasks while minimizing the average loss difference between the merged model and each individual task model. |
Unified Speech-Text Pre-training for Speech Translation and Recognition (2022.acl-long)
Copied to clipboard
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino
| Challenge: | Existing methods to pre-train speech and text use unlabeled data to learn universal feature representations. |
| Approach: | They propose a method to jointly pre-train speech and text in an encoder-decoder modeling framework for speech translation and recognition. |
| Outcome: | The proposed method achieves between 1.7 and 2.3 BLEU improvement above the state of the art on the MuST-C speech translation dataset and comparable WERs to wav2vec 2.0 on the Librispeech speech recognition task. |
Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays (2022.naacl-main)
Copied to clipboard
| Challenge: | In automatic essay grading, essay traits are important for scoring the essay holistically . a single-task learning system gives the best results for scoring essays holistically and scoring essay traits. |
| Approach: | They propose a way to score essays using a multi-task learning approach . they compare the MTL-based BiLSTM system to a single-task Learning approach based on LSTMs and BiLStms . |
| Outcome: | The proposed system gives better results for scoring essay holistically and scoring essay traits. |
Coupled Hierarchical Transformer for Stance-Aware Rumor Verification in Social Media Conversations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to rumor verification and stance classification fail to exploit intertask dependencies . |
| Approach: | They propose a Hierarchical Transformer model which uses BERT to obtain thread representations . they propose 'coupled' transformer modules to capture intertask interactions and a post-level attention layer to use predicted stance labels for RV. |
| Outcome: | The proposed model outperforms existing methods on two benchmark datasets. |
Improving End-to-End Task-Oriented Dialog System with A Simple Auxiliary Task (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Using large pre-trained language models for end-to-end TOD modeling has made significant progress on benchmarks . a paradigm of leveraging large pretrained models has shown promising results . |
| Approach: | They combine paradigm of leveraging large pre-trained language models with multi-task learning framework . their model achieves new state-of-the-art results with combined scores of 108.3 and 107.5 . |
| Outcome: | The proposed model achieves state-of-the-art results on multiWOZ 2.0 and MultiWOZ 2.1 . it also improves generalization capability through domain adaptation experiments in the few-shot setting. |
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn general representations of text can achieve sub-optimal performance in low-resource scenarios. |
| Approach: | They propose to use language model pre-training and multi-task learning to learn robust representations but these methods can achieve sub-optimal performance in low-resource scenarios. |
| Outcome: | The proposed model outperforms strong baselines on the GLUE benchmark and can be adapted to new tasks efficiently and effectively. |
Quality Estimation-Assisted Automatic Post-Editing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing APE and QE combination strategies have not shown significant performance gains in the field of automatic post-editing (APE). |
| Approach: | They propose to train a model on APE and QE tasks to improve the APE performance by using a multi-task learning methodology that treats both tasks as a 'bargaining game' they also investigate various existing combination strategies and show that their approach achieves state-of-the-art performance for a ‘distant’ language pair, viz., English-Marathi. |
| Outcome: | The proposed model improves on two different language pairs, viz., English-Marathi and English-German. |
Dynamically Updating Event Representations for Temporal Relation Classification with Multi-category Learning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models with independent classifiers for each TLINK category hinder from using the whole data. |
| Approach: | They propose a temporal relation classification model that manages dynamic event representations across multiple TLINKs using multi-task learning to leverage the full size of data. |
| Outcome: | The proposed model outperforms state-of-the-art models and two strong transfer learning baselines on English and Japanese data. |
The Discussion Tracker Corpus of Collaborative Argumentation (2020.lrec-1)
Copied to clipboard
| Challenge: | The Discussion Tracker corpus is an annotated dataset of transcripts of spoken, multi-party argumentation transcribed from 985 minutes of audio . |
| Approach: | They analyze 29 multi-party arguments transcribed from 985 minutes of audio . they provide descriptive statistics and code for predicting each dimension separately. |
| Outcome: | The Discussion Tracker corpus was collected in high school English classes and annotated for argument moves, specificity, specificities and collaboration dimensions. |
Killing Four Birds with Two Stones: Multi-Task Learning for Non-Literal Language Detection (C18-1)
Copied to clipboard
| Challenge: | idioms and metaphors are often studied in isolation, challenging the distinction . e.g., metaphorical concept mappings are ubiquitous in everyday life, thus they are ubiquitous . |
| Approach: | They propose to view the detection problem as a generalized non-literal language classification problem. |
| Outcome: | The proposed model improves on four metaphor and idiom detection tasks in two languages, English and German. |
Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models for KG-to-text generation are based on pretrained language models. |
| Approach: | They propose to automatically generate a text that describes the facts in knowledge graph (KG) they leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation. |
| Outcome: | The proposed model outperforms all comparison methods on fully-supervised and fewshot settings. |
Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on sentence prediction tasks uses shallow neural networks to learn essay representations and constrain calculated scores with regression loss or ranking loss. |
| Approach: | They propose to use a pre-trained language model to learn text representations first and then to constrain the scores with regression loss or ranking loss. |
| Outcome: | The proposed model outperforms state-of-the-art models on the Automated Student Assessment Prize dataset. |
Multi-Granularity Guided Fusion-in-Decoder (2024.findings-naacl)
Copied to clipboard
| Challenge: | Open-domain question answering requires deriving factual responses without explicit evidence . recent approaches combine retrieval of relevant information with response generation . |
| Approach: | They propose a model that concatenates multiple contexts in the decoding phase . they propose MGFiD, which harmonizes passage re-ranking with sentence classification . |
| Outcome: | The proposed model outperforms existing models on Natural Questions and TriviaQA datasets . it aggregates evident sentences into an anchor vector that instructs the decoder . |
A Unified Span-Based Approach for Opinion Mining with Syntactic Constituents (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for fine-grained opinion mining (OM) are based on span-based annotations, but they are not effective. |
| Approach: | They propose a unified span-based approach for the end-to-end OM setting using syntactic constituents and multi-task learning to integrate them into the proposed model. |
| Outcome: | The proposed approach achieves significant improvements over previous work on the MPQA 2.0 dataset and reduces the number of wrongly-predicted opinion expressions and roles. |
XLTime: A Cross-Lingual Knowledge Transfer Framework for Temporal Expression Extraction (2022.findings-naacl)
Copied to clipboard
| Challenge: | Temporal Expression Extraction (TEE) is essential for understanding time in natural language. |
| Approach: | They propose a framework for multilingual Temporal Expression Extraction that leverages pre-trained language models to prompt cross-language knowledge transfer from English to non-English languages. |
| Outcome: | The proposed framework outperforms the existing SOTA methods on French, Spanish, Portuguese, and Basque by large margins. |
McCrolin: Multi-consistency Cross-lingual Training for Retrieval Question Answering (2024.findings-emnlp)
Copied to clipboard
Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Lalita Lowphansirikul, Potsawee Manakul, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Existing approaches struggle with consistency across multiple languages and multi-size input scenarios. |
| Approach: | They propose a cross-lingual training framework that leverages multi-task learning to enhance cross-linguistic consistency and ranking stability. |
| Outcome: | The proposed training framework outperforms competitors on various input sizes and architectures. |
MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning (2024.emnlp-main)
Copied to clipboard
Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Wang Zihan, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, Libin Yang
| Challenge: | Recent advances in open-source Large Language Models (LLMs) have achieved notable successes in natural language processing. |
| Approach: | They propose a Parameter Efficient Fine-Tuning paradigm for improved fine-tuning and parameter efficiency in multi-task learning. |
| Outcome: | The proposed model outperforms existing methods on multi-task learning while reducing training costs by over 80% without losing general capability. |
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to augment retrieval-augmented generation models with retrievers often rely on spurious cues or generate hallucinations during inference. |
| Approach: | They propose a method to incorporate evidentiality of passages into training a retrieval-augmented generation model. |
| Outcome: | The proposed method outperforms its direct counterpart on all knowledge-intensive tasks. |
A Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for building multimodal open-domain dialogue agents based on large datasets are limited in real-world settings . |
| Approach: | They propose a new learning strategy called vision-language warm-up tasks for multimodal dialogue models that relies solely on learning from target data. |
| Outcome: | The proposed learning strategy achieves comparable and in some cases superior performance compared to existing state-of-the-art models on various evaluation metrics. |
UEGP: Unified Expert-Guided Pre-training for Knowledge Rekindle (2024.findings-naacl)
Copied to clipboard
Yutao Mou, Kexiang Wang, Jianhe Lin, Dehong Ma, Jun Fan, Daiting Shi, Zhicong Cheng, Gu Simiu, Dawei Yin, Weiran Xu
| Challenge: | Existing paradigms for pre-training and fine-tuning have limitations . knowledge rekindle aims to break through performance upper bounds of experts without introducing additional annotated data. |
| Approach: | They propose a new paradigm for pre-training and fine-tuning that aims to re-incorporate the fine- tuned expert model into the training cycle and break through performance upper bounds of experts. |
| Outcome: | The proposed model breaks through performance upper bounds of experts without additional annotated data. |
Learning Relatedness between Types with Prototypes for Relation Extraction (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing datasets have no intrinsic Ontology for relation types. |
| Approach: | They propose to use prototypical examples to represent each relation type and use them to augment related types from a different dataset. |
| Outcome: | The proposed model improves on a baseline with multi-task learning between datasets to obtain better representation for relations. |
Multi-Task Learning of Pairwise Sequence Classification Tasks over Disparate Label Spaces (N18-1)
Copied to clipboard
| Challenge: | Multi-task learning and semi-supervised learning are successful paradigms for learning in scenarios with limited labelled data. |
| Approach: | They propose to induce a joint embedding space between disparate label spaces and learning transfer functions between label embeddments to leverage unlabelled data and auxiliary, annotated datasets. |
| Outcome: | The proposed approach outperforms strong single and multi-task baselines and achieves state of the art on aspect-based and topic-based sentiment analysis. |
Word Emotion Induction for Multiple Languages as a Deep Multi-Task Learning Problem (N18-1)
Copied to clipboard
| Challenge: | a recent shift towards expressive emotion representation models has hampered deep learning in sentiment analysis. |
| Approach: | They propose a multi-task learning problem to solve a language data bottleneck . they propose to use word emotion induction as an individual task to predict emotion . |
| Outcome: | The proposed model outperforms a wide range of other methods on 9 languages and 15 conditions. |
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter-efficient fine-tuning are limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). |
| Approach: | They propose a parameter-efficient framework that reduces trainable parameters through tensor-train decomposition. |
| Outcome: | The proposed methods achieve comparable or better performance than most widely used methods with up to 100 fewer parameters on the LLaMA-2-7B models. |
Hierarchical Document Refinement for Long-context Retrieval-augmented Generation (2025.acl-long)
Copied to clipboard
Jiajie Jin, Xiaoxi Li, Guanting Dong, Yuyao Zhang, Yutao Zhu, Yongkang Wu, Zhonghua Li, Ye Qi, Zhicheng Dou
| Challenge: | Real-world RAG applications often encounter long-context input scenarios where redundant information and noise results in higher inference costs and reduced performance. |
| Approach: | They propose an efficient plug-and-play refiner that leverages the structural characteristics of long documents. |
| Outcome: | Experiments on seven QA datasets show that LongRefiner achieves competitive performance in various scenarios while using 10x fewer computational costs and latency compared to baseline. |
Language Modelling as a Multi-Task Problem (2021.eacl-main)
Copied to clipboard
| Challenge: | Using multitask learning, humans are optimising their behaviour towards a multitude of objectives to reach their goals in dayto-day life. |
| Approach: | They propose to study language modelling as a multi-task problem by examining the generalisation behaviour of language models as they learn the linguistic concept of Negative Polarity Items. |
| Outcome: | The proposed model is able to learn the linguistic concept of Negative Polarity Items (NPIs) and is a multi-task learning model. |
Transfer and Multi-Task Learning for Noun–Noun Compound Interpretation (D18-1)
Copied to clipboard
| Challenge: | In computational linguistics, nounnoun compound interpretation is approached as an automatic classification problem. |
| Approach: | They empirically evaluate the utility of transfer and multi-task learning on a challenging semantic classification task. |
| Outcome: | The proposed methods improve the accuracy of a neural classifier and its F1 scores on the less frequent, but more difficult relations. |
TableGPT: Few-shot Table-to-Text Generation with Table Structure Reconstruction and Content Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can produce informative and fluent text with the help of large-scale datasets, but they suffer insufficient learning problem with limited training data. |
| Approach: | They propose to use table transformation module with template to rewrite structured table in natural language as input for GPT-2 and exploit multi-task learning with two auxiliary tasks to preserve table’s structural information. |
| Outcome: | The proposed model outperforms existing systems on most few-shot settings. |
BiKT: Enabling Bidirectional Knowledge Transfer Between Pretrained Models and Sequential Downstream Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks adapt from initial pretrained model to each downstream task directly, but ignore sequential nature of downstream tasks and feedback effect on pretrained models. |
| Approach: | They propose a framework to enable bidirectional knowledge transfer between pretrained models and downstream tasks in rounds. |
| Outcome: | The proposed framework improves on 9 GLUE datasets and 6 SuperGLUEs. |
Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. |
| Approach: | They aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning by increasing the number of tasks and incorporating smaller sets of related tasks. |
| Outcome: | The proposed model improves on unseen target tasks by increasing the scale of multi-task learning to incorporate more tasks and developing similarity metrics to incorporate tasks related to the target task. |
Generalizing Natural Language Analysis through Span-relation Representations (2020.acl-main)
Copied to clipboard
| Challenge: | a large number of natural language processing tasks are generated with specially designed architectures. |
| Approach: | They propose to represent a wide variety of tasks in a single unified format . they perform extensive experiments to demonstrate benefits of multi-task learning . |
| Outcome: | The proposed model performs comparable to state-of-the-art models on 10 tasks . it also shows that it can analyze differences and similarities in how the model handles different tasks compared to other models . |
A Multi-Task Embedder For Retrieval Augmented LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing retrieval methods face limitations in terms of knowledge, memory, and action. |
| Approach: | They propose a retrieval enhancement mechanism that brings in useful information from external sources to augment the LLM. |
| Outcome: | The proposed method significantly improves the LLM’s performance in various downstream tasks while introducing superior retrieval augmentation’s effect over both general and task-specifc retrievers. |
A Gradient Control Method for Backdoor Attacks on Parameter-Efficient Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Tuning (PET) fine-tunes pre-trained language models for downstream tasks, but a large reduction in the number of attackable parameters will greatly affect the effectiveness of backdoor attacks, resulting in backdoor forgetting. |
| Approach: | They propose a gradient control method to consolidate the attack effect by freezing most parameters of the pre-trained model and fine-tuning only a small number of parameters. |
| Outcome: | The proposed method improves sentiment classification and spam detection, and can be applied to different tasks. |
Zero-Shot Rationalization by Multi-Task Transfer Learning from Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract rationales from input text are difficult and impractical. |
| Approach: | They propose a method that leverages multi-task learning and transfer learning to generate rationales through question answering in a zero-shot fashion. |
| Outcome: | The proposed method achieves comparable or even better performance without supervised signal for two benchmark rationalization datasets. |
Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neurons (2025.coling-main)
Copied to clipboard
| Challenge: | Despite their superior multitask capabilities, the multitask learning mechanisms of large language models remain as an open question. |
| Approach: | They propose a method that fine-tunes current task-specific neurons during continuous learning by using gradient attribution on task-specified data. |
| Outcome: | The proposed method is highly correlated with the given task and solves two common problems in multi-task learning and continuous learning: Generalization and Catastrophic Forgetting. |
SharpSeq: Empowering Continual Event Detection through Sharpness-Aware Sequential-task Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for continual event detection suffer from catastrophic forgetting . a novel continual learning paradigm leveraging sharpness-aware minimization is needed . |
| Approach: | They propose a continual learning paradigm that leverages sharpness-aware minimization and a generative model to balance training data distribution. |
| Outcome: | The proposed approach outperforms existing methods on real-world datasets. |
Parameter-efficient Modularised Bias Mitigation via AdapterFusion (2023.eacl-main)
Copied to clipboard
Deepak Kumar, Oleg Lesota, George Zerveas, Daniel Cohen, Carsten Eickhoff, Markus Schedl, Navid Rekabsaz
| Challenge: | Large pre-trained language models contain societal biases and carry along these biase . Current approaches to mitigate these bias impose debiasing by updating model parameters, effectively transferring model to irreversible debiased state. |
| Approach: | They propose to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand while keeping the core model untouched. |
| Outcome: | The proposed approach improves or maintains effectiveness of bias mitigation, avoids catastrophic forgetting in a multi-attribute scenario, and maintains on-par task performance while granting parameter-efficiency and easy switching between the original and debiased models. |
MetaICL: Learning to Learn In Context (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models can do in-context learning by conditioning on a few training examples with no parameter updates or task-specific templates. |
| Approach: | They propose a meta-training framework where a pretrained language model is tuned to do in-context learning on a large set of training tasks. |
| Outcome: | The proposed framework outperforms baseline models on 142 NLP datasets and a range of target tasks with domain shifts. |
Enhancing Knowledge Selection for Grounded Dialogues via Document Semantic Graphs (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing conversation models treat knowledge selection as a sentence ranking problem where each sentence is handled individually, ignoring the internal semantic connection between sentences. |
| Approach: | They propose to automatically convert background knowledge documents into document semantic graphs and perform knowledge selection over such graphs. |
| Outcome: | The proposed model improves on the knowledge selection task and the response generation task on HollE and generalizes on unseen topics in WoW. |
A Multi-Task Learning Framework for Multi-Target Stance Detection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models fail to learn target-specific representations and are prone to overfitting. |
| Approach: | They propose a multi-task learning network to train one model on all target pairs . their results show that their proposed model outperforms the best-performing baseline by 12.39% . |
| Outcome: | The proposed model outperforms the best-performing baseline model by 12.39% in macro-averaged F1-score. |
Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions (2020.acl-main)
Copied to clipboard
| Challenge: | a growing demand for the ability to communicate in English means automated tutoring and assessment systems are becoming more popular. |
| Approach: | They propose to use automatic speech recognition transcripts to grade spontaneous speech based on textual features. |
| Outcome: | The proposed system improves on a transformer encoder with native language identification as an auxiliary task. |
Do Text-to-Text Multi-Task Learners Suffer from Task Conflict? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multi-task learning architectures learn a single model across multiple tasks through a shared encoder followed by task-specific decoders. |
| Approach: | They propose to use a shared encoder and language model decoder to learn a single model across multiple tasks. |
| Outcome: | The proposed architecture does surprisingly well across a range of diverse tasks. |
Cross-domain Named Entity Recognition via Graph Matching (2022.findings-acl)
Copied to clipboard
| Challenge: | Empirical results show that our method outperforms a series of transfer learning, multitask learning, and few-shot learning methods due to the data scarcity in the real-world scenario. |
| Approach: | They propose to model the label relationship as a probability distribution and construct label graphs in both source and target label spaces. |
| Outcome: | Empirical results show that the proposed method outperforms transfer learning, multi-task learning, and few-shot learning methods on four datasets. |
CEM: Machine-Human Chatting Handoff via Causal-Enhance Module (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to predict chatbot failure ignore causal variables, resulting in cost increasement and prediction bias. |
| Approach: | They propose a machine-human chatting handoff module that predicts chatbot failure . they use user state and labor cost to correct the prediction bias . |
| Outcome: | The proposed method improves the performance of existing methods without any elaborate model crafting. |
Bits and Pieces: Investigating the Effects of Subwords in Multi-task Parsing across Languages and Domains (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural parsing is dependent on the underlying language model, but little is known about how choices affect parser performance. |
| Approach: | They examine how subword sharing is responsible for gains or negative transfer in multi-task learning . they find a preference for averaged or last subwords across languages and domains . |
| Outcome: | The proposed model favors averaged or last subwords across languages and domains . specific POS tags may require different subword, and distribution overlap is more important than discrepancies in the data sizes. |
Modeling Personal Biases in Language Use by Inducing Personalized Word Embeddings (N19-1)
Copied to clipboard
| Challenge: | Existing studies have attempted to personalize models to improve performance on NLP tasks such as sentiment analysis but they did not estimate subjective input. |
| Approach: | They propose a method of modeling personal biases in word meanings with personalized word embeddings by solving a task on subjective text while regarding words used by different individuals as different words. |
| Outcome: | The proposed method improves sentiment analysis and target task with reviews retrieved from RateBeer. |
Semi-Supervised Sequence Modeling with Cross-View Training (D18-1)
Copied to clipboard
| Challenge: | Unsupervised representation learning algorithms such as word2vec and ELMo only learn from task-specific labeled data during the main training phase. |
| Approach: | They propose a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. |
| Outcome: | The proposed algorithm improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. |
Entity Relation Extraction as Dependency Parsing in Visually Rich Documents (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on key information extraction from visually rich documents focus on labeling the text within bounding boxes, while relations between words are unexplored. |
| Approach: | They propose to use a dependency parsing model to extract semantic entities from visually rich documents by combining entity labeling and relation extraction tasks. |
| Outcome: | The proposed model achieves 65.96% F1 score on the FUNSD dataset. |
Training ELECTRA Augmented with Multi-word Selection (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-training methods for NLP tasks require massive computation resources. |
| Approach: | They propose a method that trains a discriminator to detect replaced tokens and select original tokens from candidate sets. |
| Outcome: | The proposed method improves ELECTRA based on multi-task learning on GLUE and SQUAD datasets. |
Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction (P19-1)
Copied to clipboard
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, Junji Tomita
| Challenge: | Question answering (QA) using textual sources for purposes such as reading comprehension has attracted much attention. |
| Approach: | They propose a Query Focused Extractor model for evidence extraction and multi-task learning with the QA model. |
| Outcome: | The proposed model achieves state-of-the-art evidence extraction score on hotpotQA and FEVER, which is a recognizing textual entailment task on a large textual database. |
Bridging the Code Gap: A Joint Learning Framework across Medical Coding Systems (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for automating medical coding focus on a single coding system . however, there are still challenges to overcome in coding. |
| Approach: | They propose a joint learning framework for Across Medical coding systems which jointly learns different coding system through multi-task learning. |
| Outcome: | The proposed framework improves the performance of the MIMIC-IV ICD-9 and MIMICIV I CD-10 datasets. |
Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models (2024.emnlp-main)
Copied to clipboard
Zhengxuan Wu, Yuhao Zhang, Peng Qi, Yumo Xu, Rujun Han, Yian Zhang, Jifan Chen, Bonan Min, Zhiheng Huang
| Challenge: | Modern language models fail to follow human instructions while being faithful . a trade-off exists between instruction following and faithfulness when training LMs . |
| Approach: | They propose a method that relies on Reject-sampling by Self-instruct with Continued Fine-tuning to train LMs to follow human instructions while being faithful. |
| Outcome: | The proposed method outperforms vanilla MTL with high-quality data, but with significantly smaller data. |
Semi-supervised Domain Adaptation for Dependency Parsing (P19-1)
Copied to clipboard
| Challenge: | Currently, most studies on cross-domain parsing focus on unsupervised domain adaptation . however, unsupervised approaches make limited progress due to the intrinsic difficulty of both domain adaptation and parse. |
| Approach: | They propose a semi-supervised domain adaptation problem for Chinese dependency parsing by using newly-annotated large-scale domain-aware datasets. |
| Outcome: | The proposed method is more effective than direct corpus concatenation and multi-task learning. |
Emotion-Infused Models for Explainable Psychological Stress Detection (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines the use of emotion detection for detecting psychological stress in online posts . traditional multi-task learning and emotion-based language model fine-tuning are used to improve the model . |
| Approach: | They propose to use a semantically related task, emotion detection, for detecting psychological stress in online posts . they propose multi-task learning and emotion-based language model fine-tuning to improve the model . |
| Outcome: | The proposed model is more explainable and human-like than a black-box model . the proposed model mirrors psychological components of stress, the authors show . |
Multi-Task Semantic Dependency Parsing with Policy Gradient for Learning Easy-First Strategies (P19-1)
Copied to clipboard
| Challenge: | Existing dependency parsing algorithms do not support directed acyclic graphs . a a systole-based dependency parses sentences using binary semantic relations that are not trees . |
| Approach: | They propose an iterative predicate selection algorithm for semantic dependency parsing . they train the algorithm using multi-task learning and task-specific policy gradient training . |
| Outcome: | The proposed model achieves a new state of the art on the SemEval 2015 task 18 dataset . |
Modeling with Recurrent Neural Networks for Open Vocabulary Slots (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to filling slots that take on values from a virtually unlimited set have been lacking in the natural language area. |
| Approach: | They propose a new attention-based recurrent neural network (RNN) model that captures the concept: Understanding the role of a word may vary according to how long a reader focuses on a particular part of . sentence. |
| Outcome: | The proposed model outperforms existing models with respect to discovering ‘open-vocabulary’ slots without any external information, such as a named entity database or knowledge base. |
iACOS: Advancing Implicit Sentiment Extraction with Informative and Adaptive Negative Examples (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for extracting aspects and opinions from text are incomplete. |
| Approach: | They propose a method for extracting Implicit Aspects with Categories and Opinions with Sentiments using implicit tokens. |
| Outcome: | The proposed method outperforms baseline methods on two public benchmark datasets. |
Knowledge-Interactive Network with Sentiment Polarity Intensity-Aware Multi-Task Learning for Emotion Recognition in Conversations (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Emotion Recognition in Conversation models neglect direct utterance-knowledge interaction and use emotion-indirect auxiliary tasks to augment semantic information. |
| Approach: | They propose a Knowledge-Interactive Network with sentiment polarity intensity-aware multi-task learning which leverages both commonsense knowledge and sentiment lexicon to augment semantic information. |
| Outcome: | The proposed model shows 1.04% performance improvement over the state-of-the-art model on the IEMOCAP dataset. |
Syntactic Graph Convolutional Network for Spoken Language Understanding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on slot filling and intent detection builds joint models without prior knowledge of linguistic knowledge. |
| Approach: | They propose a joint model that integrates syntactic structure for learning slot filling and intent detection jointly. |
| Outcome: | The proposed model outperforms existing models on two public benchmark datasets and further improves on slot filling and intent detection. |
A Multi-Task Learning Framework for Modeling Engagement and Topic-Sensitive Responses in Arabic Women’s Discourse (2026.findings-eacl)
Copied to clipboard
| Challenge: | a corpus of 158k arab Facebook posts spanning women's rights, gender debates, and economic empowerment reveals patterns of public opinion that vary dramatically across regional and cultural contexts. |
| Approach: | They propose a multi-task learning framework that learns audience reaction classification and engagement magnitude regression and non-engagement detection. |
| Outcome: | The proposed model achieves a test macro-F1 of 72.4 and weighted-F1. It measures 158k posts across gender issues, legal rights advocacy, gender identity discussions, and economic empowerment. |
CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing systems that negotiate with humans have broad applications in pedagogy and conversational AI. |
| Approach: | They propose to annotate persuasion strategies and perform correlation analysis to understand how dialogue behaviors are associated with the negotiation performance. |
| Outcome: | The proposed system improves negotiation performance for all strategies labeled as skewed . the proposed system is available on github.com/kushalchawla/ . |
Modelling Context Emotions using Multi-task Learning for Emotion Controlled Dialog Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent research has tackled this task using neural generative methods by augmenting emotion classes with the input sequences. |
| Approach: | They propose to use a self-attention based encoder and a decoder with dot product attention mechanism to generate a viable response with a specified emotion. |
| Outcome: | The proposed model outperforms baselines on automatic evaluation measures such as F1 and BLEU scores, thus resulting in more fluent and adequate responses. |
A Pragmatics-Centered Evaluation Framework for Natural Language Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | a number of studies have suggested that models induce universal text representations . current benchmarks focus on semantic phenomena, so pragmatics needs to be the focus . |
| Approach: | They propose a benchmark that unites 11 pragmatics-focused evaluation datasets for English. |
| Outcome: | The proposed benchmark shows that natural language inference does not result in genuinely universal representations. |
Learning to Generate Explanation from e-Hospital Services for Medical Suggestion (2022.coling-1)
Copied to clipboard
| Challenge: | Neural models have shown remarkable success in various tasks, however, simply offering the predictions may not satisfy the requirement of end-users. |
| Approach: | They propose a novel model which generates a medical suggestion and provides an explanation as the outline of the reasoning. |
| Outcome: | The proposed model achieves promising performances in both quantitative and human evaluation. |
Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat (N19-1)
Copied to clipboard
Ravi Shekhar, Aashish Venkatesh, Tim Baumgärtner, Elia Bruni, Barbara Plank, Raffaella Bernardi, Raquel Fernández
| Challenge: | Existing systems that address the abilities that need to be put to work during conversations are lacking in terms of visual grounding. |
| Approach: | They propose a visually-grounded dialogue state encoder which integrates visual grounding with dialogue system components. |
| Outcome: | The proposed system improves the GuessWhat?! game by combining guessing and asking questions with multi-task learning. |
Normalizing Compositional Structures Across Graphbanks (2020.coling-main)
Copied to clipboard
| Challenge: | Graph-based meaning representations (MRs) exhibit structural differences that reflect different theoretical and design considerations, presenting challenges to uniform linguistic analysis and cross-framework semantic parsing. |
| Approach: | They propose a method to normalize MRs at the compositional level by linguistically-grounded rules. |
| Outcome: | The proposed method increases the match in compositional structure between MRs and improves multi-task learning in a low-resource setting. |
Detecting Bipolar Disorder from Misdiagnosed Major Depressive Disorder with Mood-Aware Multi-Task Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Bipolar Disorder (BD) is a mental disorder characterized by intense mood swings, ranging from depression to manic states. |
| Approach: | They propose to use social media data to identify BD risk in individuals misdiagnosed as MDD by multi-task learning. |
| Outcome: | The proposed approach outperforms state-of-the-art baselines and can provide insights into the impact of BD mood on future risk. |
Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are limited to low-data regimes and require learning different modules to adapt to diverse tasks. |
| Approach: | They propose a framework for parameter-efficient fine-tuning that trains modules per task . they use an instance-dense retriever and a prototypical hypernetwork to generate conditional modules . |
| Outcome: | The proposed framework outperforms existing methods on multi-task learning and few-shot transfer learning. |
Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using a neural network, large language models can be trained on multiple tasks, allowing them to perform tasks efficiently. |
| Approach: | They propose a framework that leverages a neural network to select the best dataset combinations for enhancing multi-task learning (MTL) They propose to iteratively refine the selection, greatly improving efficiency while being model-, dataset-, and domain-independent. |
| Outcome: | The proposed framework iteratively refines the selection, greatly improving efficiency, while being model-, dataset-, and domain-independent. |
Syntax-Aware Opinion Role Labeling with Dependency Graph Convolutional Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Opinion role labeling (ORL) is a fine-grained opinion analysis task . due to the scarcity of labeled data, ORL remains challenging for data-driven methods due to its complexity and complexity. |
| Approach: | They propose to integrate syntactic knowledge into ORL models by comparing and integrating different representations and using dependency graph convolutional networks to encode parser information at different processing levels. |
| Outcome: | The proposed model achieves 4.34 higher F1 score than the current state-of-the-art. |
Scheduled Multi-task Learning for Neural Chat Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve chat translation focus on modeling bilingual dialogue characteristics . however, there is insufficient data and simple joint training methods . |
| Approach: | They propose a scheduled multi-task learning framework to incorporate large-scale chat translation data into training by adding a second pre-training stage between the original pre- training and fine-tuning stages. |
| Outcome: | The proposed framework improves translation tasks on four language directions. |
Continued Pretraining for Better Zero- and Few-Shot Promptability (2022.emnlp-main)
Copied to clipboard
Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy
| Challenge: | Recent language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. |
| Approach: | They propose to use a dedicated pretraining stage to improve promptability in zero-shot settings and few-shot tuning. |
| Outcome: | The proposed method improves promptability in zero- and few-shot settings, while the existing method yields subpar performance. |
NCLS: Neural Cross-Lingual Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual summarization divide the task into two steps: summarizing and translation. |
| Approach: | They propose to integrate two related tasks into the training process of CLS under multi-task learning to improve cross-lingual summarization. |
| Outcome: | The proposed framework improves on English-to-Chinese and Chinese-to English CLS human-corrected test sets. |
Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD (2021.findings-acl)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) has shown promise in various NLP tasks such as semantic dependency parsing. |
| Approach: | They propose to use two parallel treebanks to improve parsing performance. |
| Outcome: | The proposed model is based on two parallel treebanks with similar annotation schemes but differing in linguistic annotation preferences. |
Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive machine translation models have shown significant inference speedup but suffer from inferior translation accuracy. |
| Approach: | They propose to use AT as an auxiliary task to transfer AT knowledge to NAT models by knowledge distillation. |
| Outcome: | The proposed method achieves significant improvements over baseline non-Autoregressive machine translation models on WMT14 En-De and WMT16 En-Ro datasets. |
A Speaker-Aware Co-Attention Framework for Medical Dialogue Information Extraction (2022.emnlp-main)
Copied to clipboard
Yuan Xia, Zhenhui Shi, Jingbo Zhou, Jiayu Xu, Chao Lu, Yehui Yang, Lei Wang, Haifeng Huang, Xia Zhang, Junwei Liu
| Challenge: | With the development of medical digitization, the extraction and structuring of electronic medical records (EMRs) have become challenging but fundamental tasks. |
| Approach: | They propose a speaker-aware dialogue encoder with multi-task learning which takes the speaker's identity into account and a co-attention fusion network to aggregate the utterance information. |
| Outcome: | The proposed framework outperforms the state-of-the-art methods on the public medical dialogue extraction datasets to demonstrate its superiority. |
Multitask Learning-Based Neural Bridging Reference Resolution (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for bridging references lack large corpora annotated with briding references . second challenge is different definitions of bridding used in different corpors . |
| Approach: | They propose a multi-task learning-based neural model for bridging reference resolution . they show substantial improvements of up to 8 p.p. on full briding resolution compared to previous models . |
| Outcome: | The proposed model outperforms the best reported results on full bridging resolution by up to 8 p.p. |
Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | voluminous historical records are difficult to fully utilize since they are written in ancient languages and some parts are damaged over time. |
| Approach: | They propose a multi-task learning approach to restore and translate historical documents using a self-attention mechanism. |
| Outcome: | The proposed approach improves the accuracy of the translation task over baselines without multi-task learning. |
Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing summarization methods target a specific dimension, resulting in poor quality summaries. |
| Approach: | They propose multi-objective reinforcement learning tailored to generate balanced summaries across all dimensions. |
| Outcome: | The proposed model achieves significant performance gains compared to baseline models on representative summarization datasets on four dimensions. |
Structural Supervision for Word Alignment and Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge on syntactic structure neglects the rich structural information from target tokens and the structural similarity between the source and target sentences. |
| Approach: | They propose to incorporate syntactic structure of both source and target tokens into the encoder-decoder framework, tightly correlating the internal logic of word alignment and machine translation for multi-task learning. |
| Outcome: | The proposed method outperforms baselines on four publicly available language pairs and consistently outperformed baselines in alignment accuracy and translation quality. |
HD-PiSSA: High-Rank Distributed Orthogonal Adaptation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models constrain update to low-rank subspaces, limiting expressiveness and performance. |
| Approach: | They propose a distributed PEFT approach that initializes adapters across different devices and aggregates their delta updates collectively on (W) Empirically, HD-PiSSA provides 16 higher effective updated ranks than data-parallel LoRA or PiSSA when fine-tuning on 8 GPUs with the same per-device adapter rank. |
| Outcome: | Empirically, HD-PiSSA outperforms LoRA and PiSSA in math, code, and multi-task learning tasks. |
MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing datasets labeled for one task hinder multi-task learning . task-specific data make models learn task-related leakage features rather than meaningful knowledge that could generalize to other tasks. |
| Approach: | They propose to jointly label large-scale NLP dataset MATINF . it contains 1.07 million question-answer pairs with human-labeled categories . |
| Outcome: | The proposed dataset is applicable for classification, question answering, and summarization. |
Multi-task Learning to Enable Location Mention Identification in the Early Hours of a Crisis Event (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Social media is a platform for people to share their concerns and report information as eyewitnesses of events. |
| Approach: | They propose a multi-task learning approach to leverage available annotated data for several related tasks from the crisis domain to improve performance on a main task with limited annotation. |
| Outcome: | The proposed approach improves performance on a task with limited annotated data. |
Better, Faster, Stronger Sequence Tagging Constituent Parsers (N19-1)
Copied to clipboard
| Challenge: | Existing efforts to speed up constituent parsing have focused on chart-based or shift-reduce parsers. |
| Approach: | They propose to use auxiliary losses and sentence-level fine-tuning to mitigate greedy decoding issues. |
| Outcome: | The proposed model surpasses the performance of sequence tagging constituent parsers on the English and Chinese Penn Treebank datasets and reduces their parsing time even further. |
Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations (2023.findings-acl)
Copied to clipboard
Jifan Chen, Yuhao Zhang, Lan Liu, Rui Dong, Xinchi Chen, Patrick Ng, William Yang Wang, Zhiheng Huang
| Challenge: | Existing methods for multitask learning typically use a dataset name as input prefix, which limits the effectiveness of multitask training. |
| Approach: | They propose compositional task configurations, a set of prompts prepended to the encoder to improve cross-task generalization of unified models. |
| Outcome: | The proposed model outperforms the UnifiedSKG baseline by noticeable margins in both in-domain and zero-shot settings. |
Let Modalities Teach Each Other: Modal-Collaborative Knowledge Extraction and Fusion for Multimodal Knowledge Graph Completion (2025.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on missing triples in knowledge graphs, but lack correlation between modalities. |
| Approach: | They propose a framework to foster mutual guidance and collaboration in unimodal knowledge extraction and multimodal knowledge fusion. |
| Outcome: | Extensive experiments on three real-world datasets demonstrate advantages of Moodle over state-of-the-art methods. |
Multi-Task Learning and Adapted Knowledge Models for Emotion-Cause Extraction (2021.findings-acl)
Copied to clipboard
Elsbeth Turcan, Shuai Wang, Rishita Anubhai, Kasturi Bhattacharjee, Yaser Al-Onaizan, Smaranda Muresan
| Challenge: | Detecting what emotions are expressed in text is a well-studied problem in natural language processing. |
| Approach: | They propose methods that combine common-sense knowledge with multi-task learning to perform joint emotion classification and emotion cause tagging. |
| Outcome: | The proposed models improve on both tasks when using common-sense reasoning and a multitask framework. |
STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval (2020.findings-emnlp)
Copied to clipboard
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, Nigel Collier
| Challenge: | a new news dataset targets both stance detection (SD) and fine-grained evidence retrieval (ER) . stance Detection (SD), which is a form of multitask learning, has gained increasing interest in recent work . |
| Approach: | They propose a news dataset that targets both stance detection (SD) and fine-grained evidence retrieval (ER) their dataset is an expert-annotated news dataset with 3,291 articles. |
| Outcome: | The proposed dataset is a high-quality benchmark for future research in stance detection and evidence retrieval. |
SkillSpan: Hard and Soft Skill Extraction from English Job Postings (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on Skill Extraction (SE) use crowd-sourced labels or annotations from a predefined skill inventory. |
| Approach: | They propose a dataset that contains 14.5K sentences and over 12.5K annotated spans. |
| Outcome: | The proposed model outperforms non-adapted models and single-task outperformed multi-task learning. |
Training Flexible Depth Model by Multi-Task Learning for Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that multitask learning can support decoding in 24 depth configurations and is superior to individual training. |
| Approach: | They propose to use multi-task learning to train a flexible depth model that can adapt to different depth configurations during inference. |
| Outcome: | The proposed model can support decoding in 24 depth configurations and is superior to the individual training and another flexible depth model training method——LayerDrop. |
SEE: Continual Fine-tuning with Sequential Ensemble of Experts (2025.findings-acl)
Copied to clipboard
| Challenge: | Continual fine-tuning of large language models suffers from catastrophic forgetting . some approaches use routers to assign tasks to experts, but continual learning often requires retraining . |
| Approach: | They propose a framework that integrates routing and response mechanisms within each expert . it eliminates the need for an additional router and allows each expert to decide whether a query should be handled . |
| Outcome: | The proposed framework outperforms previous approaches in continual fine-tuning . it can handle learning tasks and out-of-distribution instances, paving the way for distributed model ensembling. |
Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT (2022.lrec-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is integral to many NLP applications such as chatbots and question answering. |
| Approach: | They propose to annotate Arabic nested entities instead of flat annotations by manually annotating 550K tokens with 21 entity types including person, organization, location, event and date. |
| Outcome: | The proposed model achieved an overall micro F1-score of 0.884 and the annotation guidelines and source code are publicly available. |
Knowledge Aware Emotion Recognition in Textual Conversations via Multi-Task Incremental Transformer (2020.coling-main)
Copied to clipboard
| Challenge: | Existing models for ERTC use a few non-neutral categories to identify the emotion of each utterance. |
| Approach: | They propose a novel Knowledge Aware Incremental Transformer with Multi-task Learning to address these challenges by leveraging commonsense knowledge to leverage context. |
| Outcome: | The proposed model outperforms state-of-the-art models across five benchmark datasets. |
Knowledge Graph Enhanced Neural Machine Translation via Multi-task Learning on Sub-entity Granularity (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods to integrate knowledge graph (KG) with neural machine translation (NMT) have two problems: knowledge under-utilization and granularity mismatch. |
| Approach: | They propose a multi-task learning method on sub-entity granularity to combine machine translation and knowledge reasoning tasks. |
| Outcome: | The proposed method significantly outperforms baseline models on translation tasks and handling the entities. |
Lemmatisation & Morphological Analysis of Unedited Greek: Do Simple Tasks Need Complex Solutions? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing systems for part-of-speech tagging of unedited Greek text have outperformed traditional methods for morphological analysis and lemmatisation. |
| Approach: | They propose to combine nominal features into a single label and combine the three most distinctive features of verbs into another unified label. |
| Outcome: | The proposed models outperform traditional models in lemmatisation and morphological analysis and show that multi-task learning improves performance by transferring parameters. |
Towards Emotion-aided Multi-modal Dialogue Act Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Considerable work on Dialogue Act Classification (DAC) has been done on textual inputs. |
| Approach: | They propose to use a multimodal Emotion aware Dialogue Act dataset to explore the role of multi-modality and emotion recognition in DAC. |
| Outcome: | The proposed dataset shows that multi-modality and emotion recognition improves DAC performance compared to uni-modal and single task DAC variants. |
Cross-topic Argument Mining from Heterogeneous Sources (D18-1)
Copied to clipboard
| Challenge: | Argument mining is a core technology for automating argument search in document collections. |
| Approach: | They propose a new sentential annotation scheme that is reliably applicable by crowd workers to arbitrary Web texts. |
| Outcome: | The proposed scheme outperforms vanilla BiLSTMs in two- and three-label cross-topic settings and can be further improved by leveraging additional data for topic relevance using multi-task learning. |
Ask Question First for Enhancing Lifelong Language Learning (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to stream learning NLP tasks suffer from catastrophic forgetting and are exacerbated when the previous task’s pseudo data is insufficient. |
| Approach: | They propose to use a new data format to train pseudo questions of previous tasks to stream learning NLP tasks while retaining knowledge of previous ones. |
| Outcome: | The proposed model is more robust to sufficient and insufficient pseudo-data when the task boundary is both clear and unclear. |
Task Compass: Scaling Multi-task Pre-training with Task Prefix (2022.findings-emnlp)
Copied to clipboard
Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng
| Challenge: | Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. |
| Approach: | They propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. |
| Outcome: | The proposed model can be used as a foundation backbone for a wide range of tasks and as augmentation tool for data augmentation with complementary tasks. |
Biomedical Event Extraction as Sequence Labeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Empirical results show that BeeSL’s speed and accuracy makes it a viable approach for large-scale real-world scenarios. |
| Approach: | They propose a joint end-to-end neural information extraction model that recasts the task as sequence labeling and jointly models intermediate tasks via multi-task learning. |
| Outcome: | Empirical results show that BeeSL outperforms the current best system on the Genia 2011 benchmark by 1.57% absolute F1 score reaching 60.22% F1 . |
Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events (2020.emnlp-main)
Copied to clipboard
Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, Yaser Al-Onaizan
| Challenge: | Existing models for temporal ordering of events rely on pretrained representations, transfer and multitask learning, and self-training techniques. |
| Approach: | They propose a neural architecture and a set of training methods for ordering events by predicting temporal relations by pre-training models. |
| Outcome: | The proposed models can predict temporal relations between two pairs of events within a span of text and identify temporal relationships between them. |
A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations (2020.coling-main)
Copied to clipboard
| Challenge: | e-mail conversations have domain-agnostic and two-level dialogue act annotations . et al. (2017): a better understanding of asynchronous conversations. |
| Approach: | They present a large-scale corpus of e-mail conversations with domain-agnostic and two-level dialogue act annotations . they use ISO standard 24617-2 as the annotation scheme to annotate over 6,000 messages and 35,000 sentences . |
| Outcome: | The proposed model outperforms other neural networks but falls short of human performance. |
Compositional Semantic Parsing across Graphbanks (P19-1)
Copied to clipboard
| Challenge: | Existing semantic parsers that map sentences to graph-based meaning representations are hand-designed for specific graphbanks. |
| Approach: | They propose a compositional neural semantic parser which achieves competitive accuracies across graphbanks. |
| Outcome: | The proposed system achieves competitive accuracies across a variety of graphbanks. |
GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (2021.emnlp-main)
Copied to clipboard
| Challenge: | A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. |
| Approach: | They propose an automatic auxiliary task selection method based on gradient calculation in Transformer-based models that improves MT-DNN performance. |
| Outcome: | The proposed method improves MT-DNN performance on 8 natural language understanding (GLUE) tasks, while costing less than AUTOSEM and comparable GPU consumption. |
From Awareness to Adaptability: Enhancing Tool Utilization for Scientific Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches enhance reasoning through Chain-of-Thought, Program-ofThough, and Tool-Integration. |
| Approach: | They propose a tool-awareness training method that leverages both forward and backward data generation strategies to strengthen the model’s conscious and selective tool utilization in multi-step reasoning tasks. |
| Outcome: | The proposed method improves the model's tool utilization capabilities, including proactivity and execution success rates. |
Muppet: Massive Multi-task Representations with Pre-Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows gains from pre-training and fine-tuning that are multi-task . but it can be difficult to know which intermediate tasks will best transfer . |
| Approach: | They propose a large-scale learning stage for pre-finetuning between pre-training and fine-tun. |
| Outcome: | The proposed model improves performance on pretrained discriminators and generation models on a wide range of tasks while improving sample efficiency during fine-tuning. |
PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on financial question answering systems focus on passively responding to user queries. |
| Approach: | They propose a new dataset to facilitate conversational question answering over hybrid contexts in finance . they propose PACIFIC to combine clarification question generation and CQA . |
| Outcome: | The proposed method performs multi-task learning over all sub-tasks in PACIFIC . it incorporates a simple ensemble strategy to alleviate error propagation issue . |
Different Absorption from the Same Sharing: Sifted Multi-task Learning for Fake News Detection (D19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting fake news use shared features as complementarity features without selection. |
| Approach: | They propose a sifted multi-task learning method with a selected sharing layer for fake news detection. |
| Outcome: | The proposed method boosts the F1-score by more than 0.87%, 1.31% on two public and widely used competition datasets. |
Exploiting Narrative Context and A Priori Knowledge of Categories in Textual Emotion Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for recognizing the mental state of characters in text are limited by their use of character-specific contexts. |
| Approach: | They propose a method that encodes the preceding context of the target sentence along with the target phrase using a BERT-based text encoder. |
| Outcome: | The proposed method improves the accuracy of emotion classification by encoding the preceding context of the target sentence along with the target phrase using a BERT encoder. |
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Existing simultaneous translation methods focus on text-to-text and speech-totext translation. |
| Approach: | They propose a Simul-S2ST model that jointly learns translation and simultaneous policy in a unified framework of multi-task learning. |
| Outcome: | The proposed model can perform offline and simultaneous speech recognition, speech translation and speech synthesis via an "All-in-One" seamless model. |
monoQA: Multi-Task Learning of Reranking and Answer Extraction for Open-Retrieval Conversational Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to the Conversational Question Answering task have used multi-task learning to solve the task. |
| Approach: | They propose to use multi-task learning to improve the ORConvQA task by sharing the reranker and reader’s learned structure in a generative model. |
| Outcome: | The proposed model outperforms baseline models on the OR-QuAC and OR-CoQA datasets and significantly outperformed existing strong baseline models. |
Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders their adaptability to prompt-guided Large Language Models (LLMs). |
| Approach: | They propose a framework for unified task embedding that harmonizes task embeds from various models within a single vector space. |
| Outcome: | The proposed framework harmonizes task embeddings from various models within a single vector space. |
Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to embedding in multiparty dialogues are poor for span-based question answering (QA) |
| Approach: | They propose a novel approach to transformers that learns hierarchical representations in multiparty dialogue. |
| Outcome: | The proposed model improves on the FriendsQA dataset by 3.8% and 1.4% over the two state-of-the-art models. |
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)
Copied to clipboard
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen
| Challenge: | 'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size . |
| Approach: | They propose to train a subnetwork of 'lottery tickets' to match the full model's performance. |
| Outcome: | The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large . |
Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to integrate whole hierarchical information of senses into discourse relation representations for multi-level sense recognition ignore static hierarchic structure containing all senses and ignore hierarchically sense label sequence corresponding to each instance. |
| Approach: | They propose to use a GlObal and Local Hierarchy-aware Contrastive Framework to model two kinds of hierarchies with the aid of multi-task learning and contrastive learning to learn better representations of discourse relation relationships. |
| Outcome: | The proposed method outperforms current state-of-the-art models at all hierarchical levels on PDTB 2.0 and PDTP 3.0 datasets. |
Using Eye-tracking Data to Predict the Readability of Brazilian Portuguese Sentences in Single-task, Multi-task and Sequential Transfer Learning Approaches (2020.coling-main)
Copied to clipboard
Sidney Evaldo Leal, João Marcos Munguba Vieira, Erica dos Santos Rodrigues, Elisângela Nogueira Teixeira, Sandra Aluísio
| Challenge: | Sentence complexity assessment is a relatively new task in Natural Language Processing. |
| Approach: | They propose to use Brazilian Portuguese to evaluate sentences with linguistic features to improve readability. |
| Outcome: | The proposed model reaches the state-of-the-art for Brazilian Portuguese with 97.8% accuracy with linguistic features. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
Enhanced Facet Generation with LLM Editing (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have shown that search engines can recognize facets of a user's query. |
| Approach: | They propose to use large language models to enhance the facets of a query to generate facets from a search engine. |
| Outcome: | The proposed model can predict facets by taking only queries as input without a search engine. |
Multi-Cell Compositional LSTM for NER Domain Adaptation (2020.acl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a challenging but practical problem. |
| Approach: | They propose a multi-cell compositional LSTM structure for multi-task learning . they model each entity type using a separate cell state . |
| Outcome: | Empirical results show that the proposed method outperforms multi-task learning methods and achieves the best results. |
What can we learn from Semantic Tagging? (D18-1)
Copied to clipboard
| Challenge: | a recent study shows that multi-task learning improves performance of NLP tasks by exploiting similarities between tasks. |
| Approach: | They employ semantic tagging as an auxiliary task for three NLP tasks . they compare full neural network sharing, partial neural network shared and learning what to share . |
| Outcome: | The proposed model improves for part-of-speech tagging, universal dependency parsing and natural language inference. |
Enhancing Image-to-Text Generation in Radiology Reports through Cross-modal Multi-Task Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Image-to-text generation relies on independent models for image understanding and natural language generation, which often exhibit a semantic gap between visual and textual information. |
| Approach: | They propose a multi-task learning framework to leverage both visual and non-imaging data for generating radiology reports. |
| Outcome: | The proposed framework improves performance over single-task baselines across language generation metrics and mitigates overfitting in auxiliary tasks. |
SmartSpanNER: Making SpanNER Robust in Low Resource Scenarios (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is one of the most fundamental tasks in natural language processing. |
| Approach: | They propose a method which introduces a Named Entity Head (NEH) prediction task to SpanNER and performs multi-task learning together with task of span classification. |
| Outcome: | The proposed method improves the robustness of SpanNER in low resource scenarios on the CoNLL03, Few-NERD, GENIA and ACE05 benchmark datasets. |
Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit substantial capabilities yet face challenges such as hallucination, outdated knowledge, and untraceable reasoning processes. |
| Approach: | They propose a retrieval-augmented generation approach that leverages adaptive adversarial training to dynamically adjust the model’s training process in response to retrieval noises. |
| Outcome: | The proposed approach improves the performance of the LLaMA-2 7B model under diverse noise conditions. |
A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling (D19-1)
Copied to clipboard
| Challenge: | Semantic role labeling (SRL) aims to identify the predicate-argument structure of a sentence. |
| Approach: | They propose to use a unified span-based model for Chinese SRL as a strong baseline. |
| Outcome: | The proposed framework achieves state-of-the-art 87.54 and 88.5 F1 scores on the Chinese Proposition Bank and CoNLL-2009 datasets. |
How to Ask Good Questions? Try to Leverage Paraphrases (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods to generate human-like questions rely on paraphrases to generate good questions. |
| Approach: | They propose to integrate paraphrase knowledge into question generation to generate human-like questions by combining paraphrases with a back-translation method. |
| Outcome: | The proposed model achieves obvious performance gain over several strong baselines and human evaluation validates that it can ask questions of high quality by leveraging paraphrase knowledge. |
Linguistically-Informed Self-Attention for Semantic Role Labeling (D18-1)
Copied to clipboard
| Challenge: | Existing models of semantic role labeling use no explicit linguistic features. prior work has shown that syntax trees can dramatically improve SRL decoding. |
| Approach: | They propose a neural network model that incorporates syntax using only raw tokens . they show that LISA out-performs the state-of-the-art with contextually-encoded word representations a 1.0 F1 on newswire and 2.0 F1 in out-of domain text . |
| Outcome: | The proposed model outperforms the state-of-the-art model with word embeddings and predicted predicates. |
A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis (2020.emnlp-main)
Copied to clipboard
| Challenge: | Sentiment analysis is an increasingly popular natural language processing task in academia and industry. |
| Approach: | They propose to use category name encoding network to weaken catastrophic forgetting problem . they set both encoder and decoder shared among all categories to weaker the catastrophic forgetting problem a . |
| Outcome: | The proposed model achieves state-of-the-art on two (T)ACSA benchmark datasets. |
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing performance through increased use of expert knowledge often result in diminishing sparsity during expert selection. |
| Approach: | They propose a framework that integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. |
| Outcome: | The proposed framework outperforms existing methods under identical conditions concerning the number of experts. |
DaN+: Danish Nested Named Entities and Lexical Normalization (2020.coling-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a task of finding entities in text, such as locations, organizations, and persons. |
| Approach: | They propose a multi-domain corpus and annotation guidelines for Danish nested named entities and lexical normalization to support research on cross-lingual cross-domain learning for a less-resourced language. |
| Outcome: | The proposed model outperforms existing models on the Danish Named Entity Recognition task and shows that it is robust to domain shifts and is highly effective on the least canonical data. |
Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023.findings-emnlp)
Copied to clipboard
Muling Wu, Wenhao Liu, Jianhan Xu, Changze Lv, Zixuan Ling, Tianlong Li, Longtao Huang, Xiaoqing Zheng, Xuanjing Huang
| Challenge: | Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models. |
| Approach: | They propose a token-wise prompt tuning method that uses a bank of finer-grained soft prompt tokens to generate an instance-dependent prompt. |
| Outcome: | The proposed method performs far better than full parameter fine-tuned models and achieves state-of-the-art by tuning only 0.035% parameters on 14 datasets. |
A Multi-task Learning Framework for Quality Estimation (2023.findings-acl)
Copied to clipboard
Sourabh Deoghare, Paramveer Choudhary, Diptesh Kanojia, Tharindu Ranasinghe, Pushpak Bhattacharyya, Constantin Orăsan
| Challenge: | Conventional approaches to QE involve training separate models at different levels of granularity viz., word-level, sentence-level and document-level . |
| Approach: | They propose to train a single model for sentence-level and word-level QE tasks in a multi-task learning framework and compare them to baseline models. |
| Outcome: | The proposed model improves on the single-pair, multi-patch, and zero-shot settings. |
Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions (2020.acl-main)
Copied to clipboard
| Challenge: | Visual referring expression recognition is a task that requires natural language understanding in the context of an image. |
| Approach: | They propose to use contrastive learning and multi-task learning to increase the robustness of ViLBERT, the current state-of-the-art model for this task. |
| Outcome: | The proposed methods are 12% to 23% lower in performance than the established progress for this task. |
Facilitating Cognitive Accessibility with LLMs: A Multi-Task Approach to Easy-to-Read Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to make complex texts more accessible for people with cognitive impairments are time-consuming and resource-intensive. |
| Approach: | They propose a multi-task learning approach that trains models jointly on text summarization, text simplification, and ETR generation. |
| Outcome: | The proposed approach outperforms other approaches in in-domain settings while achieving better generalization in out-of-domain scenarios. |
Out-of-Task Training for Dialog State Tracking Models (2020.coling-main)
Copied to clipboard
Michael Heck, Christian Geishauser, Hsien-chin Lin, Nurul Lubis, Marco Moresi, Carel van Niekerk, Milica Gasic
| Challenge: | Dialog state tracking (DST) suffers from data sparsity. |
| Approach: | They utilize non-dialog data from unrelated NLP tasks to train dialog state trackers . they propose to use dialog state tracking to summarise the conversation history . |
| Outcome: | The proposed method exploits non-dialog data from unrelated NLP tasks to train dialog state trackers. |
CHEER: Centrality-aware High-order Event Reasoning Network for Document-level Event Causality Identification (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies focus on building a document-level graph for cross-sentence reasoning, but ignore important causal structures. |
| Approach: | They propose a document-level event causality identification model which annotates central events and incorporates event centrality information into the reasoning network. |
| Outcome: | The proposed model performs high-order reasoning while considering event centrality. |
Joint Alignment of Multi-Task Feature and Label Spaces for Emotion Cause Pair Extraction (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for ECPE fail to model specific features and interactive features in between, or suffer from inconsistency of label prediction. |
| Approach: | They propose to align ECPE with a feature-task alignment mechanism to model emotion-&cause-specific features and the shared interactive feature. |
| Outcome: | The proposed model outperforms existing systems on all ECA subtasks. |
Come hither or go away? Recognising pre-electoral coalition signals in the news (2021.emnlp-main)
Copied to clipboard
Ines Rehbein, Simone Paolo Ponzetto, Anna Adendorf, Oke Bahnsen, Lukas Stoetzer, Heiner Stuckenschmidt
| Challenge: | In this paper, we decompose the task of recognizing from the news coverage leading up to an election the (un)willingness of political parties to form a coalition into two related, but distinct tasks. |
| Approach: | They propose a task of recognizing from news coverage the (un)willingness of political parties to form a coalition from text and a sub-task of predicting the polarity of the signal. |
| Outcome: | The proposed approach improves over a strong monolingual transfer learning baseline. |
Learnable Dependency-based Double Graph Structure for Aspect-based Sentiment Analysis (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for aspect-based sentiment classification are susceptible to dependency tree due to noisy information and neglecting rich relation information between words. |
| Approach: | They propose a dependency-based double graph model for aspect-based sentiment classification that incorporates structure, relations and linguistic features into the sentiment text. |
| Outcome: | The proposed model is superior to state-of-the-art methods on four benchmark datasets. |
Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing CodePre-trained models struggle to generalize due to superficial mapping from source code to labels instead of understanding the root causes of code vulnerabilities. |
| Approach: | They propose a framework that integrates multi-task learning with Large Language Models to effectively mine deep-seated vulnerability features. |
| Outcome: | The proposed framework surpasses seven state-of-the-art models in effectiveness, generalization, and robustness. |
A Transformer-based Threshold-Free Framework for Multi-Intent NLU (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models for multi-intent natural language understanding mainly detect multiple intents on threshold settings. |
| Approach: | They propose a transformer-based multi-intent NLU model with multi-task learning that exploits the information of the number of multiple intents in each utterance without additional manual annotations. |
| Outcome: | The proposed model achieves superior results on two public multi-intent datasets. |
A Bag of Tricks for Dialogue Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using a pretrained sequence-to-sequence language model, we explore speaker name substitution, negation scope highlighting, multi-task learning with relevant tasks, and pretraining on in-domain data. |
| Approach: | They propose a pretrained sequence-to-sequence language model that can handle different parts of dialogue belonging to multiple speakers and combine them to produce a coherent monologue summary. |
| Outcome: | The proposed techniques outperform baseline models on a dialogue summarization dataset. |
Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on multilingual automatic post-editing systems for low-resource Indo-Aryan languages have focused on different models for different language pairs. |
| Approach: | They propose to use a multilingual automatic post-editing system to improve machine translations for low-resource Indo-Aryan languages. |
| Outcome: | The proposed model outperforms English-Hindi and English-Marathi models by 2.5 and 2.39 TER points. |
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to fine-tune deep pretrained language models face catastrophic forgetting problems. |
| Approach: | They propose a recall and learn mechanism which integrates pretraining and downstream tasks into a single mechanism. |
| Outcome: | The proposed method achieves state-of-the-art performance on the GLUE benchmark and better average performance than directly fine-tuning of BERT-large. |
Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge (2025.findings-emnlp)
Copied to clipboard
Yan-Lun Chen, Yi-Ru Wei, Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang, Ying-Dar Lin, Yu-Sung Wu, Wei-Bin Lee
| Challenge: | Large language models (LLMs) demonstrate strong task-specific capabilities through fine-tuning, but merging multiple fine- tuned models often leads to degraded performance due to overlapping instruction-following components. |
| Approach: | They propose a layer-wise approach that assigns layer-specific weights to task vectors based on their alignment with instruction-following or task-specific components. |
| Outcome: | The proposed approach outperforms existing methods in learning and forgetting tasks while preserving overall model utility. |
Multi-Task Learning for Chemical Named Entity Recognition with Chemical Compound Paraphrasing (D19-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is one of the important basic technologies for Natural Language Processing (NLP) . |
| Approach: | They propose to use long short-term memory (LSTM) of NER model to capture chemical com- pound paraphrases by sharing parameters of LSTM and character embeddings be- tween the two models. |
| Outcome: | The proposed method improves chemi- cal NER and achieves state-of-the-art performance on the BioCreative IV’s CHEMDNER task. |
Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning? (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning (PT) based on frozen pre-trained language models has shown remarkable performance in few-shot learning . however, it relies heavily on good initialization of the prompt embeddings. |
| Approach: | They propose to use meta prompt tuning to improve cross-task generalization by learning to initialize prompt embeddings from other relevant tasks. |
| Outcome: | The proposed method outperforms PT on classification tasks, but not multi-task learning. |
Rethinking and Improving Multi-task Learning for End-to-end Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | auxiliary tasks are highly consistent with end-to-end speech translation (ST) but their effectiveness has not been thoroughly studied. |
| Approach: | They propose an improved multi-task learning approach for the ST task that bridges the modal gap by mitigating the difference in length and representation. |
| Outcome: | The proposed approach achieves state-of-the-art on the MuST-C dataset with 20.8% of training time required by the current SOTA method. |
Improving Image Captioning with Better Use of Caption (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to image captioning focus on visual attention, but many do not. |
| Approach: | They propose a framework that explores semantics available in captions and leverages that to enhance both image representation and caption generation. |
| Outcome: | The proposed framework outperforms baselines on the MSCOCO dataset and is state-of-the-art under a wide range of evaluation metrics. |
Enhancing Relation Extraction via Adversarial Multi-task Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies have focused on re-modeling the given NEs and thus lead to inferior results when NE is sometimes ambiguous. |
| Approach: | They propose a relation extraction model with two training stages that uses adversarial multi-task learning to recover the given NEs. |
| Outcome: | The proposed model improves on two English benchmark datasets and shows state-of-the-art performance. |
TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition (2025.acl-long)
Copied to clipboard
Tianwei Lin, Jiang Liu, Wenqiao Zhang, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Jiannan Guo, Hao Jiang, Siliang Tang, Yueting Zhuang
| Challenge: | Existing methods for fine-tuning are resource-efficient, but performance often falls short . a new approach, TeamLoRA, integrates collaborative and competitive modules to improve performance. |
| Approach: | They propose to introduce task-specific LoRA as domain experts to improve learning efficiency . teamLoRA integrates collaborative and competition modules to improve model learning . |
| Outcome: | Experiments show that TeamLoRA improves performance in multi-task learning . teamLorea integrates collaborative and competitive modules to improve performance . |
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models for Natural Languages (NL) like BERT and GPT have been shown to transfer well to Programming Languages. |
| Approach: | They propose a unified pre-trained encoder-decoder Transformer model that leverages the code semantics conveyed from the developer-assigned identifiers. |
| Outcome: | The proposed model outperforms existing models on understanding and generation tasks and can capture semantic information from code. |
Spatial Coordinates as a Cell Language: A Multi-Sentence Framework for Imaging Mass Cytometry Analysis (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing single-cell LLMs struggle to integrate spatial information into natural language, limiting their ability to capture biological relationships. |
| Approach: | They propose a framework that integrates both single-cell expression and spatial information into natural language using a multi-sentence approach. |
| Outcome: | The proposed framework outperforms existing single-cell LLMs on preprocessed IMC datasets for diabetes and brain tumors while improving interpretability. |
Exploiting the Syntax-Model Consistency for Neural Relation Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Existing deep learning models for Relation Extraction (RE) have limited generalization beyond the syntactic structures of the input sentences. |
| Approach: | They propose a deep learning model that uses dependency trees to extract syntactic importance of words for Relation Extraction. |
| Outcome: | The proposed model outperforms existing models on three RE benchmark datasets. |
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
A Multitask Learning Approach for Diacritic Restoration (2020.acl-main)
Copied to clipboard
| Challenge: | Diacritics are used to specify pronunciations and meanings in many languages like Arabic. |
| Approach: | They propose to use multi-task learning to optimize diacritic restoration with related NLP problems . they use Arabic as a case study since it has sufficient data resources for tasks . |
| Outcome: | The proposed model outperforms baseline models and is comparable to the state-of-the-art models. |
Multi-Domain Named Entity Recognition with Genre-Aware and Agnostic Inference (2020.acl-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a key component of many text processing pipelines. |
| Approach: | They propose a new architecture tailored to the task of identifying named entities with data from multiple genres. |
| Outcome: | The proposed architecture outperforms baseline and competitive methods on all three setups with differences ranging between +1.95 to +3.11 average F1 across multiple genres when compared to standard approaches. |
TXtract: Taxonomy-Aware Knowledge Extraction for Thousands of Product Categories (2020.acl-main)
Copied to clipboard
| Challenge: | State-of-the-art methods for knowledge extraction are designed for a single category of product, but do not apply to real-life e-Commerce scenarios. |
| Approach: | They propose a taxonomy-aware knowledge extraction model that applies to thousands of categories organized in a hierarchical taxonomies. |
| Outcome: | The proposed model outperforms state-of-the-art methods on 4,000 categories in F1 and 15% across all categories. |
On the Compositional Generalization in Versatile Open-domain Dialogue (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning suffer from interference among datasets or fail to effectively reuse knowledge and skills learned from other datasets. |
| Approach: | They propose a sparsely activated modular network with a well-rounded set of operators and instantiate each operator with an independent module. |
| Outcome: | The proposed model outperforms state-of-the-art supervised approaches on 4 datasets with only 10% training data thanks to the modular architecture and multi-task learning. |
Improving Chinese Named Entity Recognition with Multi-grained Words and Part-of-Speech Tags via Joint Modeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition (CNER) is a fundamental task in natural language processing (NLP). |
| Approach: | They propose a tree parsing approach for jointly modeling Chinese named entity recognition (CNER) with multi-grained word segmentation (MWS) and POS tagging tasks. |
| Outcome: | The proposed approach achieves better or comparable performance with current methods. |
DualGuard: A Parameter Space Transformation Approach for Bidirectional Defense in Split-Based LLM Fine-Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing defense methods for large language model fine-tuning (LLM-FT) sacrifice task-specific performance under privacy constraints. |
| Approach: | They propose a bidirectional defense mechanism that uses a local warm-up parameter transformation to alter client-side model parameters before training. |
| Outcome: | The proposed defense mechanism outperforms current defense methods while maintaining task performance. |
Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge Distillation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models that use separate recommendation and dialogue modules produce inconsistent results . a multi-task learning model is proposed to bridge the gap between recommendation and generated responses . |
| Approach: | They propose a multi-task learning model that integrates knowledge from two teachers and selectively gates between them via Contextualized Knowledge Distillation. |
| Outcome: | The proposed model significantly improves recommendation performance while enhancing fluency and achieves comparable results in terms of diversity. |
Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained large language models retain task-specific knowledge, but where and to what extent they retain it remains unexplored. |
| Approach: | They investigate the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations across over 60 NLP tasks. |
| Outcome: | The results show that pre-trained models retain task-specific knowledge . some tasks are already encoded in pre-train models, but others benefit from instruction tuning. |
A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown the importance of visual information in multi-party conversations due to the complexity of visual scenes. |
| Approach: | They propose a framework to extract face sequences as visual features from a real speaker's utterance and a pipeline method to extract the face sequence. |
| Outcome: | The proposed framework extracts face sequences of the real speaker of each utterance and improves emotion prediction on the MELD dataset. |
Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Contemporary advances in NLP are built on the representational power of latent embedding spaces learned by self-supervised language models (LMs). |
| Approach: | They use a new information theoretic probing suite to analyze representational subspaces in language models. |
| Outcome: | The proposed approach compared performance of nine tasks across 2M pre-training steps and five seeds. |
Understanding and Bridging the Modality Gap for Speech Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve end-to-end speech translation (ST) use multitask learning, but there is always a modality gap between ST and MT due to the differences between speech and text. |
| Approach: | They propose a method to bridge the modality gap between ST and MT by leveraging (text) machine translation data. |
| Outcome: | The proposed method bridges the modality gap and achieves significant improvements over baseline in all eight directions. |
Stereotype Detection as a Catalyst for Enhanced Bias Detection: A Multi-Task Learning Approach (2025.findings-acl)
Copied to clipboard
| Challenge: | a new study addresses bias and stereotypes in language models by exploring how learning them together improves performance. |
| Approach: | They propose a dataset for bias and stereotype detection that integrates religion, gender, socio-economic status, race, profession, and others. |
| Outcome: | The proposed dataset compares encoder-only models and fine-tuned decoder- only models . the results show that learning stereotypes together improves bias detection . |
MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfried Kunzmann, Zheng Zhang
| Challenge: | Large language models (LLMs) have demonstrated exceptional capabilities across diverse tasks, but their fine-tuning requires significant memory, posing challenges for resource-constrained environments. |
| Approach: | They propose a ZO-based framework that eliminates the need for backpropagation and provides a memory-efficient alternative to backprograming. |
| Outcome: | The proposed framework surpasses first-order methods in performance and accuracy. |
M3TCM: Multi-modal Multi-task Context Model for Utterance Classification in Motivational Interviews (2024.lrec-main)
Copied to clipboard
| Challenge: | Motivational interviews have two distinct roles, namely client and therapist . previous approaches did not fully incorporate all of these characteristics into utterance classification . |
| Approach: | They propose a multi-modal, multi-task context model for utterance classification that integrates text and speech as well as conversation context. |
| Outcome: | The proposed model outperforms the state-of-the-art in utterance classification on the AnnoMI dataset with a relative improvement of 20% for the client- and by 15% for therapist utterrance classification. |
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification (2024.findings-emnlp)
Copied to clipboard
Zeren Shui, Petros Karypis, Daniel Karls, Mingjian Wen, Saurav Manchanda, Ellad Tadmor, George Karypis
| Challenge: | Prior research has shown that pretrained language models (PLMs) can achieve state-of-the-art performance on CIC benchmarks. |
| Approach: | They propose a multi-task learning framework that fine-tunes pretrained language models on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals. |
| Outcome: | The proposed framework outperforms current state-of-the-art models on small datasets while aligning with the best-performing model on a large dataset. |
Impartial Multi-task Representation Learning via Variance-invariant Probabilistic Decoding (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods focus on balancing loss or gradients but fail to address this issue due to the representation discrepancy in latent space. |
| Approach: | They propose a framework that harmonizes representation spaces across tasks to ensure impartial learning by harmonizing representation spaces. |
| Outcome: | The proposed framework outperforms 12 representative methods under the same multi-task settings, especially in heterogeneous task combinations and data-constrained scenarios. |
Mixture-of-LoRAs: An Efficient Multitask Tuning Method for Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models. |
| Approach: | They propose a mixture-of-LoRAs architecture which is a parameter-efficient tuning method designed for multi-task learning with LLMs. |
| Outcome: | The proposed method can be iteratively adapted to a new domain, enabling quick domain-specific adaptation. |
Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that multi-head attention module evolves functional specialization after multi-task training. |
| Approach: | They propose a method to quantify the degree of functional specialization in multi-head attention . they propose 'multi-task training' method to increase functional specialisation and mitigate negative information transfer . |
| Outcome: | The proposed method increases functional specialization and mitigates negative information transfer in multi-task learning without adding any parameters. |
MECH: A Cost-Effective Multi-Task Cascade Framework for Classroom Opinion Evolution Recognition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on dialogue act annotation, overlooking the deeper dimension of opinion evolution. |
| Approach: | They propose a framework for Classroom Opinion Evolution Recognition that translates "Action-Opinion" dualism into a risk-aware routing mechanism. |
| Outcome: | The proposed framework achieves state-of-the-art accuracy of 78.55% while reducing API costs by 44.4%. |
Reference-guided Style-Consistent Content Transfer (2024.lrec-main)
Copied to clipboard
| Challenge: | Text style transfer involves changing the style of a text while preserving its original style. |
| Approach: | They propose a task of style-consistent content transfer which involves modifying a text’s content based on a provided reference statement while preserving its original style. |
| Outcome: | The proposed approach meets three important conditions: reference faithfulness, style adherence, and coherence. |
Sebastian, Basti, Wastl?! Recognizing Named Entities in Bavarian Dialectal Data (2024.lrec-main)
Copied to clipboard
Siyao Peng, Zihang Sun, Huangyan Shan, Marie Kolm, Verena Blaschke, Ekaterina Artemova, Barbara Plank
| Challenge: | Named Entity Recognition (NER) is a fundamental task to extract key information from texts, but annotated resources are scarce for dialects. |
| Approach: | They present the first dialectal NER dataset for German, BarNER, with 161K tokens annotated on Bavarian Wikipedia articles and tweets. |
| Outcome: | The proposed dataset improves on bar-wiki and moderately on bartweet with training first on Bavarian . |
Exploring Multimodal Relation Extraction of Hierarchical Tabular Data with Multi-task Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies overlook the need of mining relations among multiple columns rather than just the semantic relation between two specific columns in real-world practice. |
| Approach: | They propose a Chain-of-Thought distillation framework with self-correction mechanism to enhance MLLMs’ reasoning capabilities without increasing parameter scale. |
| Outcome: | The proposed method significantly outperforms baselines on wide datasets. |
Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, we tried native sample mixing, multi-task learning, and prompting and instruction finetuning very large multilingual language models (VMLMs). |
| Approach: | They used native sample mixing, multi-task learning and prompting and instruction finetuning to improve code-mixed humour and sarcasm detection. |
| Outcome: | The proposed methods improve humour and sarcasm detection by adding native samples to training sets and multitask learning and prompting and instruction finetuning VMLMs. |
Just a Scratch: Enhancing LLM Capabilities for Self-harm Detection through Intent Differentiation and Emoji Interpretation (2025.acl-long)
Copied to clipboard
| Challenge: | Self-harm detection on social media is critical for early intervention and mental health support, yet remains challenging due to the subtle, context-dependent nature of such expressions. |
| Approach: | They propose a framework to distinguish intent through nuanced language–emoji interplay. |
| Outcome: | The proposed framework improves self-harm detection and explanation tasks on three state-of-the-art LLMs. |
Structure-aware Fine-tuning for Code Pre-trained Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing CodePTMs are mainly structure-free and structurebased, but how to fine-tune them remains a challenge. |
| Approach: | They propose a plug-and-play fine-tuning method that incorporates structural knowledge into pre-trained code models. |
| Outcome: | The proposed method can benefit CodePTMs more with limited training data. |
Training-free LLM Merging for Multi-task Learning (2025.acl-long)
Copied to clipboard
Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
| Challenge: | Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing tasks. |
| Approach: | They propose a training-free method for unifying different specialized LLMs into a single model using model-wise and layer-wise pruning and scaling. |
| Outcome: | The proposed method outperforms existing merging techniques and surpasses models fine-tuned on combined datasets in most scenarios. |
DrFrattn: Directly Learn Adaptive Policy from Attention for Simultaneous Machine Translation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to learn read/write policies from attention mechanism may compromise effectiveness of attention mechanism . |
| Approach: | They propose a method that directly learns adaptive policies from the attention mechanism . experimental results demonstrate that the method achieves an improved balance between translation accuracy and latency. |
| Outcome: | The proposed method achieves improved balance between translation accuracy and latency. |
Rethinking Depression Prediction from a Fine-Grained Subscore Modeling Perspective via Multi-Task Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for depression assessment rely on standardized ratings, but they are time-consuming and subject to inter-rater variability. |
| Approach: | They propose a fine-grained model for subscore prediction via multi-task learning that can be used to predict depression severity using multiple tasks. |
| Outcome: | The proposed model outperforms baselines and Qwen3-14B direct scoring on the public E-DAIC dataset and to a large-scale private clinical dataset. |
Typology-Aware Multilingual Morphosyntactic Parsing with Joint Abstract Node Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | UniDive 2025 Morphosyntactic Parsing (MSP) shared task unifies dependency structure, morphological features, and unrealized arguments. |
| Approach: | They propose a multilingual, typology-aware joint system that integrates word-type prediction, content-only parsing, morphological tagging, and an abstract-node component within a single architecture. |
| Outcome: | The proposed model outperforms the leading submission by 3.23 percentage points in MSLAS, 3.35 in LAS, and 1.78 in FEATS macro F1. |